CI: Move CircleCI job to Azure pipelines

[datapythonista]

We've been moving our CI from other systems to Travis and Azure Pipelines, which makes things simpler, as we don't need to maintain several CI configuration files, and check for CI errors and logs in different environments.

We have almost everything in those two systems now, except one last job that is in CircleCI. This job requires PostgreSQL and MySQL databases, which hasn't been configured yet in Azure Pipelines. So, some research on how to set up them will be required.

In this issue we should create a new job in Azure Pipelines ci/azure/linux.yml equivalent to the one in CircleCI, and remove all CircleCI files .circleci.

CC: @TomAugspurger

[alexander-ponomaroff]

I would like to work on this.

[datapythonista]

That would be great, thanks!

[alexander-ponomaroff]

Hey @datapythonista , I’ve been doing some research needed for this issue and I would like to ask for some tips. From what I understand in .circleci, PostgreSQL and MySQL databases are configured through docker. When I was looking into doing this in Azure Pipelines, I found documentation on how to set up MySQL and PostgreSQL natively on linux/windows/mac Azure virtual machine. Also I found a way to build images with Docker, like it is done in .circleci folder, however I need to login using id and password by running “docker login” command. Source: https://docs.microsoft.com/en-us/azure/devops/pipelines/languages/docker?view=vsts&tabs=yaml.

Sorry if I’m misunderstanding the task, this is my first time working with CI and Docker, but I would like to learn and explore this. I would be grateful for any tips you have.

[datapythonista]

Unfortunately I'm not an expert myself in Azure Pipelines or docker. Setting the db servers natively sounds good unless it's slow. If you want to explore that option, that would be great. I'd check the docker option if that one is too complex or too slow. But if you want to check it, we can see that login you need.

[alexander-ponomaroff]

@datapythonista I will try to explore both options and update you on what I decided to use. If you know somebody that may be able to give some tips on this. Please tag them if possible.

[datapythonista]

I haven't work with db tests, but to provide some context to the best of my knowledge, in pandas/tests/io/test_sql.py there are some tests that connect to the different database flavors, and that are skipped if the driver modules (e.g. pymysql, pshycopg2) are not installed.

So, they will just run in some of our builds (one of them being the one in CircleCI). I was just checking for pymysql and I see that there are actually some builds in Azure that should be running the sql tests:

$ grep "pymysql" ci/deps/*
azure-27-compat.yaml:    - pymysql==0.6.0
azure-36-locale_slow.yaml:  - pymysql
azure-37-locale.yaml:  - pymysql
circle-36-locale.yaml:  - pymysql
travis-27.yaml:  - pymysql=0.6.3
travis-36-slow.yaml:  - pymysql
travis-36.yaml:  - pymysql

I'm not quite sure if there is some other reason for which the tests that do hit the db are being skipped. Or if we already have db servers working in Azure (may be they are provided by default).

@TomAugspurger could you let us know why MySQL and PostgreSQL servers are set up in CircleCI, but not in Azure, if both seem to be running the pandas sql tests?

@alexander-ponomaroff unrelated to that, note that there are some small changes in the CircleCI config in #23866. Shouldn't affect you much, but letting you know, so you are aware of them.

[TomAugspurger]

I'm not quite sure if there is some other reason for which the tests that do hit the db are being skipped.

The tests are skipped when we can't connect to the database pandas_nosetest. I'm not sure exactly where in the class hierarchy that's done though. So I don't think all the sql tests are running on azure.

[TomAugspurger]

If you go to https://dev.azure.com/pandas-dev/pandas/_build/results?buildId=4052&view=ms.vss-test-web.test-result-details and filter the files for e.g. TestPostgreSQLAlchemy, you'll see skips like

<pandas.tests.io.test_sql.TestMySQLAlchemy object at 0x7f2eb6d6fc50> - can't connect to mysql server

[datapythonista]

That's good to know, I didn't see it when checking the code.

What I'd do for now is to move the CircleCI build to Travis, where the db is already set up. And in a separate PR may be move something that doesn't use the db from travis to azure, so travis builds finish faster.

How does this sound?

[TomAugspurger]

I don't really have a preference.

…

On Mon, Nov 26, 2018 at 9:46 AM Marc Garcia ***@***.***> wrote: That's good to know, I didn't see it when checking the code. What I'd do for now is to move the CircleCI build to Travis, where the db is already set up. And in a separate PR may be move something that doesn't use the db from travis to azure, so travis builds finish faster. How does this sound? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#23821 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABQHImd1nub2kHzV_Jit0cvlfVxGxqpMks5uzAzcgaJpZM4Yr4s4> .

[datapythonista]

@alexander-ponomaroff if what I said makes sense to you, can you give it a try? I think it'll make things simpler to have 2 CI systems instead of 3, both for the set up, and when checking the results in a PR.

Related to this PR, I created this issue: #23928

[alexander-ponomaroff]

@datapythonista I will be looking into this today and tomorrow. Thanks.