The tuple-based methods (the first 4) are a factor more efficient than pd.Series-based methods (the last 3).%timeit, row) for _, row in df].iterrows()] # 11.6 s %timeit df.apply(lambda row: divide(row, row), axis=1) # 4.83 s %timeit df.apply(lambda row: divide(*row), axis=1, raw=True) # 760 ms %timeit ].itertuples(index=False)] # 112 ms %timeit np.vectorize(divide)(df, df) # 48.1 ms %timeit list(map(divide, df, df)) # 43.9 ms For the purposes of assignment to a series within a dataframe, the results are comparable. Below are all Python-level loops which produce either pd.Series, np.ndarray or list objects containing the same values. 1 The entire point of vectorised calculations is to avoid Python-level loops by moving calculations to highly optimised C code and utilising contiguous memory blocks. I will start by saying that the power of Pandas and NumPy arrays is derived from high-performance vectorised calculations on numeric arrays. How to apply a function to two columns of Pandas dataframe How do I use Pandas 'apply' function to multiple columns? Pandas create new column based on values from other columns If np.vectorize() is in general always faster than df.apply(), then why is np.vectorize() not mentioned more? I only ever see StackOverflow posts related to df.apply(), such as: N=10000, df.apply: 1 sec, np.vectorize: 0 sec The results are shown below: N=1000, df.apply: 0 sec, np.vectorize: 0 sec # Make sure results from df.apply and np.vectorize match.Īssert(df.equals(df)) Print 'N=%d, df.apply: %d sec, np.vectorize: %d sec' % \ Result_vectorize = end_epoch_sec - start_epoch_sec Result_apply = end_epoch_sec - start_epoch_secĭf = np.vectorize(divide)(df, df) Is this an expected result, and why?įor example, suppose I have the following dataframe with N rows: N = 10ĭf = pd.DataFrame()ĭf = df.apply(lambda row: divide(row, row), axis=1) From what I measured (shown below in some experiments), using np.vectorize() is 25x faster (or more) than using the DataFrame function apply(), at least on my 2016 MacBook Pro.
I have not seen a good discussion of the speed difference between df.apply() and np.vectorize(), so I thought I would ask here. I am using Pandas dataframes and want to create a new column as a function of existing columns.
0 kommentar(er)
