Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

Not currently. But I imagine that, if Pandas does adopt Arrow in its next version, it should be able to do something like that through proper use of the Arrow API. Arrow is built with this kind of processing in mind and is continually adding more compute kernels that work this way when possible. The Dataset abstraction in Arrow allows for defining complex column "projections" that can execute in a single pass like this. Polars may be leveraging this functionality in Arrow.


Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: