Catching Elusive Joe is the main quest given by the Jungle Fisherman. Once you acquired the fisherman's set, you need to catch the rarest fish in the jungle waters, Elusive Joe.
To start the quest, you need to equip the Fisherman's Boots, Fisherman's Pants and Fisherman's Vest. Talk to the Fisherman and he will give you clues about the Pink Fishing Lure, necessary to catch Elusive Joe. The lure can be crafted with a Spinning Bucktail Lure (acquired from a merchant quest) and a Red Fishing Lure, or an Exquisite Deep Sea Lure. The former can be bought from merchants, while the latter can be found in abandoned campsite crates.
Once you have the pink lure, equip it and start fishing on jungle islands. The pink lure has a fishing malus, so it's recommended to equip any item that increase fishing skills and to drink a Sea Fisher's Brew. Keep in mind the pink fishing lure can be consumed, so it's recommended to save once you got it and reset the game in case it broke.
Once you finally managed to catch Elusive Joe, return to the Fisherman to give him the fish in exchange for the Lucky Fisherman's Hat.