No module named psycopg2 mac os

ModuleNotFoundError: No module named ‘psycopg2._psycopg’ #51

Comments

soft2dev commented Sep 25, 2019

When I deploy it, I get this error.

I checked _psycopg module but there is no this module.
I used psycopg2-3.7

I hope your help.

The text was updated successfully, but these errors were encountered:

salerak commented Oct 1, 2019 •

how are you building and deploying this? because you’ll need to copy the right folder depending on the version for example, I’m using the ssl version 3.7 and I’m doing this
mv awslambda-psycopg2/with_ssl_support/psycopg2-3.7/ ./v-env/lib/python3.7/site-packages/psycopg2 . Make sure you’re also using python version 3.7

jayshah96 commented Oct 11, 2019 •

facing the same issue. I tried without SSL for python 3.7.

gffield commented Oct 15, 2019

Getting this issue as well with psycopg2-3.6

genii90 commented Nov 12, 2019

On a windows 10 machine. I am getting the same error. I copied psycopg2-3.7 to my root directory and renamed the folder to psycopg2. I package and deploy my application using «sam» commands. My Lambda environment is set to Python 3.7. Are there any special steps when using a Windows environment?

genii90 commented Nov 14, 2019

I ended up using a different library: psycopg2-binary in my requirement.txt file and it working fine now.

evohnave commented Nov 21, 2019

Has anyone been able to get the 3.7 files to work with Lambda? I got 3.6 to work (make sure your files are in ./python/psycopg2 and zip the python directory before creating your layer) but couldn’t get 3.7 to work at all and get the psycopg2._psycopg missing error.

shlomi-viz commented Jan 21, 2020

any updates on getting it to work with Python 3.7?

shlomi-viz commented Jan 22, 2020

I solved it by using psycopg2-binary==2.8.3

Marco-Rod commented Jan 23, 2020

I solved it by using psycopg2-binary==2.8.3

which python version?

shlomi-viz commented Jan 24, 2020

I solved it by using psycopg2-binary==2.8.3

which python version?

matthewhegarty commented Apr 17, 2020 •

I couldn’t get it to work with a requirements.txt file.

The only thing that worked for me was to statically include dependencies in the same directory as the handler (I used the psycopg2-3.6 lib), but 3.7 works as well:

Simple Makefile (include the static libraries in a static_deps dir):

sinjihn-4th-valley commented Jun 11, 2020 •

im having this issue just trying to run locally. Python = 3.7.6. I copied the python 3.7 folder.

Jugbot commented Sep 11, 2020

Was this issue solved?

genii90 commented Sep 11, 2020

I didn’t solve it. I found a work around by using psycopg2-binary instead. Not sure that solution is sufficient for you.

NedyalkoKr commented Oct 13, 2020

On Windows 10 downgrading to psycopg2-binary==2.8.5 with python 3.8.6 works and django. I don’t know why 2.8.6 doesn’t work.

fluxquantum commented Oct 30, 2020 •

I solved it by using psycopg2-binary==2.8.3

Was this from doing a pip install as part of deployment or using lambda layers? I ideally want to y. use that version of psycopg2-binary, but just settled with using psycopg2 offered in this repo for now.

fluxquantum commented Oct 30, 2020

Has anyone been able to get the 3.7 files to work with Lambda? I got 3.6 to work (make sure your files are in ./python/psycopg2 and zip the python directory before creating your layer) but couldn’t get 3.7 to work at all and get the psycopg2._psycopg missing error.

To create a layer — I had to create a folder of this path: /python/lib/python3.7/site-packages/psycopg2

shlomi-viz commented Oct 31, 2020

I use it as a regular package and install it with pip.

danielkuske commented Apr 4, 2021

I was facing the same error after I moved the psycopg2 folder from the root level of my project to a subfolder (package/psycopg2). It seems, that several import statements inside psycopg2 do not work after that.
My intention was to have a cleaner structure. I finally achieved that by uploading psycopg2 to a lambda layer (which makes the lambda function even cleaner).

If you want to do so, just put the psycopg2 folder into a folder called python, zip it and upload it to a new lambda layer. This lambda layer can now be used by every lambda function that needs psycopg2.

Источник

Lambda Import Error: No module named ‘psycopg2._psycopg’ #47

Comments

brentonmallen1 commented Aug 2, 2019 •

I tried to both use the compiled binary as well as using the following the manual build instruction but I still get a No module named ‘psycopg2._psycopg’ error on the lambda function. It seem like there is a _psycopg module that is attempting to be imported here:

but that file doesn’t seem to exist in the compiled directory. Am I missing something?

I’ve also gone ahead and dockerized the build process, I was going to submit it in a PR with some instructions in the readme if that would be useful:

Also, I’m aware of this issue: #23 but seeing as I’m having the issue on lambda, and the change noted above is recent, I figured I’d make an issue.

The text was updated successfully, but these errors were encountered:

kumprj commented Aug 6, 2019 •

Also experiencing this issue. Tried compiling in python 2.7 and 3.6 (and also python 3.7) using @jkehler ‘s psycopg2 folders (with the respective rename). Maybe I’m missing something with the pre-requisities but every step works in this tutorial except for Step 5.5 where you invoke the lambda function.
aws apigateway get-rest-apis generates an api but it gets an internal server error when trying to curl query it. I do think the issue lies with just this lambda/psycopg2 step.

edit: also tried sudo yum install python-psycopg2 and no luck after zipping that way either. Also tried using an s3 bucket made by my lambda/cloudformation.

cklyyung commented Aug 7, 2019

When you added the psycopg2 folders to the lambda package, were they placed in python/lib/python3.7/site-packages ? I was having also an issue with the No module named error, and I had to change the folder structure to follow what was documented here to fix it.

brentonmallen1 commented Aug 7, 2019 •

I was able to get this to work today. My use case was using this as a lambda layer in a custom directory path. I was initially trying to add the path using sys.path.append() but that didn’t work. Turns out adding the path to the psycopg2 folder using sys.path.insert() worked.

[updated] os.path.append() to sys.path.append()

kumprj commented Aug 7, 2019

@brentonmallen1 can you elaborate a little more on that?

@cklyyung yeah I moved the files there there manually so I don’t think that’s it :\

brentonmallen1 commented Aug 7, 2019

my use case was to use a lambda layer. To do that, I had to
zip up the contents of the psycopg2 directory (after compiling) and submit it to aws.

When a lambda function is invoked, it will unpack that zip file to the /opt directory. The different thing in my case was that I had a parent directory in which the psycopg2 directory was contained. So the unpacked path to the module looks something like:

In order for the lambda function to be able to import the module, I had to append the PYTHONPATH environment variable with the path to the module. I originally tried doing that using the sys.path.append(‘/opt/parent’) (I mistyped earlier) method at the beginning of the function script, but that didn’t work as it doesn’t append the PYTHONPATH per se. In order to get it to work, I had to use the sys.path.insert(0, ‘/opt/parent’) method at the beginning of the function script.

I hope that helps, if not let me know and I’ll see how I can help.

kumprj commented Aug 10, 2019 •

serverless-query.zip
I’m uploading this zip file. However, I am having trouble understanding the layers element.

kumprj commented Aug 10, 2019

When you added the psycopg2 folders to the lambda package, were they placed in python/lib/python3.7/site-packages ? I was having also an issue with the No module named error, and I had to change the folder structure to follow what was documented here to fix it.

I have no layers at the moment, maybe adding them will help.

brentonmallen1 commented Aug 10, 2019

@kumprj if you don’t want to use lambda layers, the psycopg2/ directory needs to be in the same directory as your lambda handler script. That way the import is relative to the current directory. So the when the function zip file is unzipped, the structure should look something like:

If you’re going to try to put it in a lambda layer, the zipped library needs to be of the same path structure as what you want it to be on the lambda function. For example, the file structure in the zip folder, when unzipped, should look like this:

Also, ensure that your runtime environment is indeed python3.7 otherwise the directory might not be there.

This path only works because the site-packages path is automatically added to the PYTHONPATH in the lambda function.

kumprj commented Aug 10, 2019 •

psycopg2 and the lambda function.py are in the same directory. I appreciate your help but I’m clearly off the mark somewhere. I’m open to using the lambda layers but this is my first time using lambda so I’m not sure what I’m doing.

I’m using python 2.7 and I made sure the template was using python2.7 too, so that should all be right.

brentonmallen1 commented Aug 10, 2019

It could be as annoying as a typo somewhere. Double check the handler call in the aws console for the lambda function in question and see if it matches the name of your script exactly.

kumprj commented Aug 11, 2019 •

Now its getting Unable to import module ‘serverless-query’: libpq.so.5: cannot open shared object file: No such file or directory which I thought this repo was supposed to solve, so I need to check out where my gap is there.

brentonmallen1 commented Aug 12, 2019

@kumprj another potential route to take would be taking advantage of Serverless to handle the library packaging: https://twitter.com/tdhopper/status/1159111484925984769?s=20

oynix commented Sep 17, 2019 •

I did as README.md said but it was not worked and that made me confused.
But now, I may be know what problem was.

After you compiled on your own computer, there was a so file in the directory named psycopg2 in psycopg2-2.6.1/build/lib.xxxxxx/ .

Python import so library by the file name that called ELF header depend on runtime environment. The name of so file after I compiled on my Mac is psycopg.cpython-37m-darwin.so , it was suitable for MaxOS, but AWS Lambda run on linux.

So, when I deploy my code on AWS Lambda, it always report that No module named ‘psycopg2._psycopg’ .

Resolution I found was that redo the step 1/2/3/4 in README.md on a Linux machine, after compile I got a so file named _psycopg.cpython-37m-x86_64-linux-gnu.so in directory psycopg2 .

When I copied this directory into zip file and redeployed my code to Lambda, it worked.

BTW, psycopg2-2.8.3 is a better choice rather than psycopg2-2.6.1 .

coltonbh commented Mar 19, 2020

I can confirm that building using the psycopg2-binary on a Mac system and then uploading that .zip file to lambda results in the missing import. Building on a linux system and uploading the .zip file will solve the missing import problem.

brentonmallen1 commented Mar 19, 2020

This is most likely because lambda run on Amazon Linux which is a CentOS variant. My usual course of action is to build the dependencies in an Amazon Linux Docker container and then deploy the package with those dependencies.

rodrich commented Jan 25, 2021

This is most likely because lambda run on Amazon Linux which is a CentOS variant. My usual course of action is to build the dependencies in an Amazon Linux Docker container and then deploy the package with those dependencies.

Yes, you right! Thanks!. I had same issue on my macOS System and I was able to solve it by creating an Amazon Linux EC2 Instance based on Amazon Linux AMI to lambda runtimes. Check below:

After that, I followed great contribution of @jkehler and I downloaded and compiled from scratch:
https://github.com/jkehler/awslambda-psycopg2

kumprj commented Jan 25, 2021

This is most likely because lambda run on Amazon Linux which is a CentOS variant. My usual course of action is to build the dependencies in an Amazon Linux Docker container and then deploy the package with those dependencies.

Yes, you right! Thanks!. I had same issue on my macOS System and I was able to solve it by creating an Amazon Linux EC2 Instance based on Amazon Linux AMI to lambda runtimes. Check below:

After that, I followed great contribution of @jkehler and I downloaded and compiled from scratch:
https://github.com/jkehler/awslambda-psycopg2

rodrich commented Jan 25, 2021

This is most likely because lambda run on Amazon Linux which is a CentOS variant. My usual course of action is to build the dependencies in an Amazon Linux Docker container and then deploy the package with those dependencies.

Yes, you right! Thanks!. I had same issue on my macOS System and I was able to solve it by creating an Amazon Linux EC2 Instance based on Amazon Linux AMI to lambda runtimes. Check below:
https://docs.aws.amazon.com/lambda/latest/dg/lambda-runtimes.html
https://console.aws.amazon.com/ec2/v2/home#Images:visibility=public-images;search=amzn-ami-hvm-2018.03.0.20181129-x86_64-gp2
After that, I followed great contribution of @jkehler and I downloaded and compiled from scratch:
https://github.com/jkehler/awslambda-psycopg2

In my case, Pip package doesn’t work, for that reason I had to use a workaround.

rodrich commented Jan 25, 2021

This is most likely because lambda run on Amazon Linux which is a CentOS variant. My usual course of action is to build the dependencies in an Amazon Linux Docker container and then deploy the package with those dependencies.

Yes, you right! Thanks!. I had same issue on my macOS System and I was able to solve it by creating an Amazon Linux EC2 Instance based on Amazon Linux AMI to lambda runtimes. Check below:
https://docs.aws.amazon.com/lambda/latest/dg/lambda-runtimes.html
https://console.aws.amazon.com/ec2/v2/home#Images:visibility=public-images;search=amzn-ami-hvm-2018.03.0.20181129-x86_64-gp2
After that, I followed great contribution of @jkehler and I downloaded and compiled from scratch:
https://github.com/jkehler/awslambda-psycopg2

In my case, Pip package doesn’t work, for that reason I had to use a workaround. Remember that It just happened at time to create and deploy aws lambda, locally It works fine and no issue.

Источник