move the max threads and fetch limit in fetcher into env update the postprocess flow to remove duplicates