There are many different types of minnows in North America, but the one you are most likely to encounter kayaking is the Golden Shiner, and there is a good reason for this: If you are out after dark and you have a light, they will jump at it and can end up in the boat.
The Swimming River is an extension of the Navesink. On the map above, you can see the main put-in is at the upper-right in the new (2015) county park. You need to check in at the ranger's hut; there is a ramp fee of $7 per vessel, including kayaks and canoes. The park also has a strict closing time, at which the gates are locked. For details, see the county's page here: