How to tackle a large number of records using Python dataframes in Odoo?
Tips for efficient data processing in Odoo

Working with Python Dataframes in Odoo

  by Hafiz Junaid

When you are working with large numbers of records in Odoo, it can be helpful to use Python dataframes to process and analyse the data more efficiently. Here are some tips for working with dataframes in Odoo:

1. Use the read_group method

The read_group method can be used to retrieve aggregated data from Odoo models in a single query. This can be especially useful when dealing with large numbers of records. You can then convert the result into pandas dataframe using the pd.DataFrame method.

2. Use the search method with chunking

If you need to retrieve a large number of records, you can use the search method to retrieve the records in chunks instead of all at once. This can help to reduce memory usage and improve performance. You can do this by specifying the offset and limit parameters to retrieve the records in batches.

3. Use the drop_duplicates method

If you have duplicate records in your data, you can use the drop_duplicates method to remove them. This can help to reduce the size of your dataframe and make it easier to work with.

4.  Use the apply method 

The apply method can be used to apply a function to each row or column of a dataframe. This can be useful for performing calculations or transformations on your data.

5.  Use the merge method 

The merge method can be used to combine data from multiple dataframes based on a common column. This can be useful for combining data from different models in Odoo. 

By using these tips and techniques, you can work more efficiently with large numbers of records in Odoo using Python dataframes.

Have a Look at the Example Script

Here's an example script for working with large numbers of records in Odoo using Python dataframes:

from odoo import models, fields, api

import pandas as pd


class MyModel(models.Model):

    _name = 'my.model'

 

    @api.model

    def process_data(self):

        # Use read_group to retrieve aggregated data

        data = self.read_group([], ['field1', 'field2', 'field3'], ['field1', 'field2'])

       

        # Convert result into a pandas dataframe

        df = pd.DataFrame(data)

       

        # Use search with chunking to retrieve records

        records = self.search([], offset=0, limit=1000)

        while records:

            # Process the records

            df = df.append(records.read(), ignore_index=True)

            # Get the next batch of records

            offset += 1000

            records = self.search([], offset=offset, limit=1000)

       

        # Use drop_duplicates to remove duplicates

        df = df.drop_duplicates()

       

        # Use apply to perform calculations on data

        df['field4'] = df.apply(lambda row: row['field1'] + row['field2'], axis=1)


        # Use merge to combine data from multiple dataframes

        other_data = pd.DataFrame(self.env['other.model'].search([]).read())

        merged_data = pd.merge(df, other_data, on='common_field')

       

        # Use write to save the data back to Odoo

        for index, row in merged_data.iterrows():

            record = self.search([('field1', '=', row['field1'])])

            record.write({'field2': row['field2'], 'field3': row['field3'], 'field4': row['field4']})

In the above script, the process_data method uses the read_group method to retrieve aggregated data from the model. It then uses the search method with chunking to retrieve the records in batches and process them. The drop_duplicates method is used to remove duplicates, and the apply method is used to perform calculations on the data. The merge method is used to combine data from multiple dataframes, and the write method is used to save the data back to Odoo.

Note that this is just an example and may need to be modified to fit your specific use case. Additionally, you may want to consider using other libraries or techniques such as lazy loading or database indexing to further optimise your code for large data sets.
Set Up Country-Specific Price Lists on Your Odoo Website