We should change how duplicated works

[OrestZborowski-SIG]

I would like to propose a specific change to the duplicated method in rt_fastarray. In particular, there is a kwarg high_unique that I think was clearly intended to be passed into Grouping (as the kwarg lex) which speeds up the process for arrays with many unique values by a lot by doing a sort to get nCountGroup or whatever. So I would propose something like this

    def duplicated(self, keep='first', high_unique=False):
        '''
        See pandas.Series.duplicated

        Duplicated values are indicated as True values in the resulting
        FastArray. Either all duplicates, all except the first or all except the
        last occurrence of duplicates can be indicated.

        Parameters
        ----------
        keep : {'first', 'last', False}, default 'first'
            - 'first' : Mark duplicates as True except for the first occurrence.
            - 'last' : Mark duplicates as True except for the last occurrence.
            - False : Mark values with just one occurrence as False.

        high_unique : bool, default False
            Change this to true if your array has many unique values

        '''
        arr = self

        if keep == 'last':
            arr = arr[::-1].copy()

        elif keep is not False and keep != 'first':
            raise ValueError(f'keep must be either "first", "last" or False')

        # create an return array all set to True
        result = ones(len(arr), dtype=np.bool_)

        g = Grouping(arr._fa if hasattr(arr,'_fa') else arr, lex = high_unique)

        if keep is False:
            # search for groups with a count of 1
           result[g.ifirstkey[g.ncountgroup[1:]==1]] = False
        else:
            result[g.ifirstkey] = False

            if keep == 'last':
                result= result[::-1].copy()
        return result

Although obviously feel free to change anything, including the comments. Over all, we definitely could have a more efficient algorithm here (you don’t have to essentially create a categorical just to find duplicated elements, and even if you do create a categorical, I think we probably could be doing both the searching and hashing in parallel which would significantly speed up a lot of operations that we do) but I think we should at a minimum allow people to input a high_unique kwarg. Thanks