Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Initial IMPORT FOREIGN SCHEMA support #62

Merged
merged 2 commits into from
Jul 27, 2015

Conversation

jjthiessen
Copy link
Contributor

This provides basic/initial support for IMPORT FOREIGN SCHEMA in PostgreSQL 9.5+

@jjthiessen jjthiessen mentioned this pull request Jun 26, 2015
@ahsanhadi
Copy link

Thanks. I plan to test this feature next week.
On Jun 26, 2015 11:49 PM, "jjthiessen" notifications@github.com wrote:

This provides basic/initial support for IMPORT FOREIGN SCHEMA in

PostgreSQL 9.5+

You can view, comment on, or merge this pull request online at:

#62
Commit Summary

  • Update to reflect changes in PostgreSQL headers (9.5 compatibility)
  • Add initial IMPORT FOREIGN SCHEMA support

File Changes

Patch Links:


Reply to this email directly or view it on GitHub
#62.

scan_relid,
params_list,
fdw_private);
return make_foreignscan(tlist
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Moving the commas to the start of the line is not common to the PostgreSQL code base and is unrelated to the feature.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I switched the formatting around to allow for capturing the change (within the #if) as locally as possible while maintaining consistent formatting across the items in the argument list of make_foreignscan.

I'm also fine with only using a start of line comma on the line within the #if (maintaining previous formatting/trailing commas elsewhere); or encapsulating the whole return in a #if/#else macro block (possibly cleaner looking, but more duplicated lines).

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am getting the error below when trying to compile the mysql_fdw with PG
9.5.

Ibrar is going to commit a fix for this later today.

make USE_PGXS=1
gcc -Wall -Wmissing-prototypes -Wpointer-arith
-Wdeclaration-after-statement -Wendif-labels -Wmissing-format-attribute
-Wformat-security -fno-strict-aliasing -fwrapv -fexcess-precision=standard
-O2 -fpic -I/home/edb/mysql-5.6.19-linux-glibc2.5-x86_64/include -D
_MYSQL_LIBNAME="libmysqlclient.so" -I. -I./
-I/home/edb/Projects/PG_servers/postgres/binary/include/server
-I/home/edb/Projects/PG_servers/postgres/binary/include/internal
-D_GNU_SOURCE -c -o connection.o connection.c
gcc -Wall -Wmissing-prototypes -Wpointer-arith
-Wdeclaration-after-statement -Wendif-labels -Wmissing-format-attribute
-Wformat-security -fno-strict-aliasing -fwrapv -fexcess-precision=standard
-O2 -fpic -I/home/edb/mysql-5.6.19-linux-glibc2.5-x86_64/include -D
_MYSQL_LIBNAME="libmysqlclient.so" -I. -I./
-I/home/edb/Projects/PG_servers/postgres/binary/include/server
-I/home/edb/Projects/PG_servers/postgres/binary/include/internal
-D_GNU_SOURCE -c -o option.o option.c
gcc -Wall -Wmissing-prototypes -Wpointer-arith
-Wdeclaration-after-statement -Wendif-labels -Wmissing-format-attribute
-Wformat-security -fno-strict-aliasing -fwrapv -fexcess-precision=standard
-O2 -fpic -I/home/edb/mysql-5.6.19-linux-glibc2.5-x86_64/include -D
_MYSQL_LIBNAME="libmysqlclient.so" -I. -I./
-I/home/edb/Projects/PG_servers/postgres/binary/include/server
-I/home/edb/Projects/PG_servers/postgres/binary/include/internal
-D_GNU_SOURCE -c -o deparse.o deparse.c
gcc -Wall -Wmissing-prototypes -Wpointer-arith
-Wdeclaration-after-statement -Wendif-labels -Wmissing-format-attribute
-Wformat-security -fno-strict-aliasing -fwrapv -fexcess-precision=standard
-O2 -fpic -I/home/edb/mysql-5.6.19-linux-glibc2.5-x86_64/include -D
_MYSQL_LIBNAME="libmysqlclient.so" -I. -I./
-I/home/edb/Projects/PG_servers/postgres/binary/include/server
-I/home/edb/Projects/PG_servers/postgres/binary/include/internal
-D_GNU_SOURCE -c -o mysql_query.o mysql_query.c
gcc -Wall -Wmissing-prototypes -Wpointer-arith
-Wdeclaration-after-statement -Wendif-labels -Wmissing-format-attribute
-Wformat-security -fno-strict-aliasing -fwrapv -fexcess-precision=standard
-O2 -fpic -I/home/edb/mysql-5.6.19-linux-glibc2.5-x86_64/include -D
_MYSQL_LIBNAME="libmysqlclient.so" -I. -I./
-I/home/edb/Projects/PG_servers/postgres/binary/include/server
-I/home/edb/Projects/PG_servers/postgres/binary/include/internal
-D_GNU_SOURCE -c -o mysql_fdw.o mysql_fdw.c
mysql_fdw.c: In function ‘mysqlGetForeignPlan’:
mysql_fdw.c:949:8: error: too few arguments to function ‘make_foreignscan’
In file included from mysql_fdw.c:41:0:
/home/edb/Projects/PG_servers/postgres/binary/include/server/optimizer/planmain.h:46:21:
note: declared here
mysql_fdw.c: In function ‘mysqlPlanForeignModify’:
mysql_fdw.c:1073:35: error: ‘RangeTblEntry’ has no member named
‘modifiedCols’
mysql_fdw.c: In function ‘mysqlGetForeignPlan’:
mysql_fdw.c:950:1: warning: control reaches end of non-void function
[-Wreturn-type]
make: *** [mysql_fdw.o] Error 1

On Sat, Jun 27, 2015 at 4:18 AM, jjthiessen notifications@github.com
wrote:

In mysql_fdw.c
#62 (comment):

@@ -942,11 +950,15 @@ mysqlGetForeignPlan(PlannerInfo *root,RelOptInfo *baserel, Oid foreigntableid, F
* field of the finished plan node; we can't keep them in private state
* because then they wouldn't be subject to later planner processing.
*/

  • return make_foreignscan(tlist,
  •                       local_exprs,
    
  •                       scan_relid,
    
  •                       params_list,
    
  •                       fdw_private);
    
  • return make_foreignscan(tlist

I switched the formatting around to allow for capturing the change (within
the #if) as locally as possible while maintaining consistent formatting
across the items in the argument list of make_foreignscan.

I'm also fine with only using a start of line comma on the line within the
#if (maintaining previous formatting/trailing commas elsewhere); or
encapsulating the whole return in a #if/#else macro block (possibly
cleaner looking, but more duplicated lines).


Reply to this email directly or view it on GitHub
https://github.com/EnterpriseDB/mysql_fdw/pull/62/files#r33404303.

Ahsan Hadi
Snr Director Product Development
EnterpriseDB Corporation
The Enterprise Postgres Company

Phone: +92-51-8358874
Mobile: +92-333-5162114

Website: www.enterprisedb.com
EnterpriseDB Blog: http://blogs.enterprisedb.com/
Follow us on Twitter: http://www.twitter.com/enterprisedb

This e-mail message (and any attachment) is intended for the use of the
individual or entity to whom it is addressed. This message contains
information from EnterpriseDB Corporation that may be privileged,
confidential, or exempt from disclosure under applicable law. If you are
not the intended recipient or authorized to receive this for the intended
recipient, any use, dissemination, distribution, retention, archiving, or
copying of this communication is strictly prohibited. If you have received
this e-mail in error, please notify the sender immediately by reply e-mail
and delete this message.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This branch is based on my PG_95 branch which should have fixed the compile time errors that you listed (please see commit 5f8e57c). What branch are you on, and is 5f8e57c in the commit log?

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks.

I was able to apply your patch and get the import foreign schema working
successfully. However I am running into crash when i try to select from any
of the mysql foreign table.

It seem to be crashing in the external library. Is anyone else seeing this
issue?

postgres=# select * from test.emp;
server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
The connection to the server was lost. Attempting reset: Failed.

(gdb) bt
#0 0x00007f8243a7c037 in raise () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f8243a7f698 in abort () from /lib/x86_64-linux-gnu/libc.so.6
#2 0x00000000007d7327 in ExceptionalCondition ()
#3 0x000000000061ad47 in list_delete ()
#4 0x00007f82399c31d4 in mysql_stmt_close (stmt=0x1d07de0)
at
/export/home/pb2/build/sb_0-12116699-1399374319.34/mysql-5.6.19/libmysql/libmysql.c:4705
#5 0x00007f8239f5d38a in mysqlEndForeignScan () from
/home/edb/Projects/PG_servers/postgres/binary/lib/mysql_fdw.so
#6 0x00000000005ffa5e in ExecEndForeignScan ()
#7 0x00000000005db9d6 in standard_ExecutorEnd ()
#8 0x00000000005a5203 in PortalCleanup ()
#9 0x00000000007fbd13 in PortalDrop ()
#10 0x00000000006e895c in PostgresMain ()
#11 0x00000000004654b3 in ServerLoop ()
#12 0x000000000068a6de in PostmasterMain ()
#13 0x00000000004660c6 in main ()
(gdb)

-- Ahsan

On Thu, Jul 2, 2015 at 9:55 PM, jjthiessen notifications@github.com wrote:

In mysql_fdw.c
#62 (comment):

@@ -942,11 +950,15 @@ mysqlGetForeignPlan(PlannerInfo *root,RelOptInfo *baserel, Oid foreigntableid, F
* field of the finished plan node; we can't keep them in private state
* because then they wouldn't be subject to later planner processing.
*/

  • return make_foreignscan(tlist,
  •                       local_exprs,
    
  •                       scan_relid,
    
  •                       params_list,
    
  •                       fdw_private);
    
  • return make_foreignscan(tlist

This branch is based on my PG_95 branch which should have fixed the
compile time errors that you listed (please see commit 5f8e57c
5f8e57c).
What branch are you on, and is 5f8e57c
5f8e57c
in the commit log?


Reply to this email directly or view it on GitHub
https://github.com/EnterpriseDB/mysql_fdw/pull/62/files#r33798674.

Ahsan Hadi
Snr Director Product Development
EnterpriseDB Corporation
The Enterprise Postgres Company

Phone: +92-51-8358874
Mobile: +92-333-5162114

Website: www.enterprisedb.com
EnterpriseDB Blog: http://blogs.enterprisedb.com/
Follow us on Twitter: http://www.twitter.com/enterprisedb

This e-mail message (and any attachment) is intended for the use of the
individual or entity to whom it is addressed. This message contains
information from EnterpriseDB Corporation that may be privileged,
confidential, or exempt from disclosure under applicable law. If you are
not the intended recipient or authorized to receive this for the intended
recipient, any use, dissemination, distribution, retention, archiving, or
copying of this communication is strictly prohibited. If you have received
this e-mail in error, please notify the sender immediately by reply e-mail
and delete this message.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tested without the patch, the crash is not reproducible.

Can the author of the patch please look into this issue...I will do further
testing on import foreign schema once this issue is fixed...

On Sat, Jul 4, 2015 at 1:28 PM, Ahsan Hadi ahsan.hadi@enterprisedb.com
wrote:

Thanks.

I was able to apply your patch and get the import foreign schema working
successfully. However I am running into crash when i try to select from any
of the mysql foreign table.

It seem to be crashing in the external library. Is anyone else seeing this
issue?

postgres=# select * from test.emp;
server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
The connection to the server was lost. Attempting reset: Failed.

(gdb) bt
#0 0x00007f8243a7c037 in raise () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007f8243a7f698 in abort () from /lib/x86_64-linux-gnu/libc.so.6
#2 0x00000000007d7327 in ExceptionalCondition ()
#3 0x000000000061ad47 in list_delete ()
#4 0x00007f82399c31d4 in mysql_stmt_close (stmt=0x1d07de0)
at
/export/home/pb2/build/sb_0-12116699-1399374319.34/mysql-5.6.19/libmysql/libmysql.c:4705
#5 0x00007f8239f5d38a in mysqlEndForeignScan () from
/home/edb/Projects/PG_servers/postgres/binary/lib/mysql_fdw.so
#6 0x00000000005ffa5e in ExecEndForeignScan ()
#7 0x00000000005db9d6 in standard_ExecutorEnd ()
#8 0x00000000005a5203 in PortalCleanup ()
#9 0x00000000007fbd13 in PortalDrop ()
#10 0x00000000006e895c in PostgresMain ()
#11 0x00000000004654b3 in ServerLoop ()
#12 0x000000000068a6de in PostmasterMain ()
#13 0x00000000004660c6 in main ()
(gdb)

-- Ahsan

On Thu, Jul 2, 2015 at 9:55 PM, jjthiessen notifications@github.com
wrote:

In mysql_fdw.c
#62 (comment):

@@ -942,11 +950,15 @@ mysqlGetForeignPlan(PlannerInfo *root,RelOptInfo *baserel, Oid foreigntableid, F
* field of the finished plan node; we can't keep them in private state
* because then they wouldn't be subject to later planner processing.
*/

  • return make_foreignscan(tlist,
  •                      local_exprs,
    
  •                      scan_relid,
    
  •                      params_list,
    
  •                      fdw_private);
    
  • return make_foreignscan(tlist

This branch is based on my PG_95 branch which should have fixed the
compile time errors that you listed (please see commit 5f8e57c
5f8e57c).
What branch are you on, and is 5f8e57c
5f8e57c
in the commit log?


Reply to this email directly or view it on GitHub
https://github.com/EnterpriseDB/mysql_fdw/pull/62/files#r33798674.

Ahsan Hadi
Snr Director Product Development
EnterpriseDB Corporation
The Enterprise Postgres Company

Phone: +92-51-8358874
Mobile: +92-333-5162114

Website: www.enterprisedb.com
EnterpriseDB Blog: http://blogs.enterprisedb.com/
Follow us on Twitter: http://www.twitter.com/enterprisedb

This e-mail message (and any attachment) is intended for the use of the
individual or entity to whom it is addressed. This message contains
information from EnterpriseDB Corporation that may be privileged,
confidential, or exempt from disclosure under applicable law. If you are
not the intended recipient or authorized to receive this for the intended
recipient, any use, dissemination, distribution, retention, archiving, or
copying of this communication is strictly prohibited. If you have received
this e-mail in error, please notify the sender immediately by reply e-mail
and delete this message.

Ahsan Hadi
Snr Director Product Development
EnterpriseDB Corporation
The Enterprise Postgres Company

Phone: +92-51-8358874
Mobile: +92-333-5162114

Website: www.enterprisedb.com
EnterpriseDB Blog: http://blogs.enterprisedb.com/
Follow us on Twitter: http://www.twitter.com/enterprisedb

This e-mail message (and any attachment) is intended for the use of the
individual or entity to whom it is addressed. This message contains
information from EnterpriseDB Corporation that may be privileged,
confidential, or exempt from disclosure under applicable law. If you are
not the intended recipient or authorized to receive this for the intended
recipient, any use, dissemination, distribution, retention, archiving, or
copying of this communication is strictly prohibited. If you have received
this e-mail in error, please notify the sender immediately by reply e-mail
and delete this message.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have yet to be able to reproduce the crash that you're reporting (I've admittedly done very limited testing). Could you please provide the SQL that you used when you encountered the crash (on both the MySQL and PostgreSQL sides)? Could you also please provide the PostgreSQL commit that you were on? Any other details about your build environment would be appreciated.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On Tue, Jul 7, 2015 at 9:33 PM, jjthiessen notifications@github.com wrote:

In mysql_fdw.c
#62 (comment):

@@ -942,11 +950,15 @@ mysqlGetForeignPlan(PlannerInfo *root,RelOptInfo *baserel, Oid foreigntableid, F
* field of the finished plan node; we can't keep them in private state
* because then they wouldn't be subject to later planner processing.
*/

  • return make_foreignscan(tlist,
  •                       local_exprs,
    
  •                       scan_relid,
    
  •                       params_list,
    
  •                       fdw_private);
    
  • return make_foreignscan(tlist

I have yet to be able to reproduce the crash that you're reporting (I've
admittedly done very limited testing). Could you please provide the SQL
that you used when you encountered the crash (on both the MySQL and
PostgreSQL sides)? Could you also please provide the PostgreSQL commit that
you were on? Any other details about your build environment would be
appreciated.

I can reproduce the crash consistently using the following steps :

1- Compiled and installed mysql fdw by pulling the latest code from github
2- Applied the attached patch for import foreign schema functionality and
the import schema was done fine
3- Tried to create another mysql fdw foreign table and did select from it
and it crashed.
4- Crash is not reproducible without the patch

I am using PG 9.5 alpha and mysql 5.6.19.


Reply to this email directly or view it on GitHub
https://github.com/EnterpriseDB/mysql_fdw/pull/62/files#r34060458.

Ahsan Hadi
Snr Director Product Development
EnterpriseDB Corporation
The Enterprise Postgres Company

Phone: +92-51-8358874
Mobile: +92-333-5162114

Website: www.enterprisedb.com
EnterpriseDB Blog: http://blogs.enterprisedb.com/
Follow us on Twitter: http://www.twitter.com/enterprisedb

This e-mail message (and any attachment) is intended for the use of the
individual or entity to whom it is addressed. This message contains
information from EnterpriseDB Corporation that may be privileged,
confidential, or exempt from disclosure under applicable law. If you are
not the intended recipient or authorized to receive this for the intended
recipient, any use, dissemination, distribution, retention, archiving, or
copying of this communication is strictly prohibited. If you have received
this e-mail in error, please notify the sender immediately by reply e-mail
and delete this message.

@jjthiessen jjthiessen force-pushed the import-foreign-schema branch from e2fad16 to e25d2b8 Compare July 8, 2015 18:37
@jjthiessen
Copy link
Contributor Author

Amended commit to use ANSI quoting of string literals.

@jjthiessen
Copy link
Contributor Author

@ahsanhadi I tried to replicate the crash, but I've been unable to do so thus far. However, there are potentially significant differences between our environments:

  • I'm running MySQL 5.5 and I haven't setup a proper build env to test against different versions
  • I'm using the current HEAD of PostgreSQL's REL9_5_STABLE git branch (it sounded like you might be using the alpha1 tarball)
  • I'm merging my mysql_fdw:import-foreign-schema branch into an ephemeral fork of the official mysql_fdw:master to simulate the merge that would actually happen (I got the impression that you were manually applying patches or cherry-picking based on your earlier comments)).

I did encounter an issue with how I was quoting string literals (that would cause imports to fail (without crashing) depending on execution order). This particular quoting bug should be fixed in my most recent push.

While you'll have to amend the SQL test code and login parameters to match your server, I've used something like the following to bootstrap, build, and test (in reality, I generally use separate scripts for each component and I don't do clean builds each time I test, but I have run the following in my environment (with different SQL matching my MySQL setup), and it executes without the crash that you mention):

#!/bin/bash

# Directory structure setup
BASE=${1:-${HOME}/src/import-foreign-schema-test}
mkdir -p ${BASE} \
    && cd ${BASE} \
    || exit

# PostgreSQL setup
if [[ ! -d "${BASE}/postgres" ]]; then
    git clone https://github.com/postgres/postgres.git ${BASE}/postgres
fi

cd ${BASE}/postgres \
    && git checkout REL9_5_STABLE \
    && git pull \
    || exit

HASH=$(git rev-parse HEAD)
HASHDIR=${BASE}/pg/${HASH}

export PGPORT=9595
export PGDATA=${HASHDIR}/data
export PATH=${HASHDIR}/bin:${PATH}

./configure --prefix=${HASHDIR} \
    && make clean \
    && make \
    && make check install \
    || exit

if [[ ! -d "${PGDATA}" ]]; then
    initdb
fi

pg_ctl -l ${HASHDIR}/pg.log start

# mysql_fdw setup
if [[ ! -d "${BASE}/mysql_fdw" ]]; then
    git clone https://github.com/EnterpriseDB/mysql_fdw.git ${BASE}/mysql_fdw
    git remote add jjthiessen https://github.com/jjthiessen/mysql_fdw.git
fi

cd ${BASE}/mysql_fdw \
    && git checkout master \
    && git pull \
    && (git branch -D ephemeral; git checkout -b ephemeral) \
    && git fetch --all \
    && git merge jjthiessen/import-foreign-schema \
    && (rm *.o *.so; make USE_PGXS=1) \
    && make USE_PGXS=1 install \
    || exit

# mysql_fdw test cases
psql -d postgres <<SQL
\timing

show server_version_num;

drop database if exists mysql_fdw-test;
create database mysql_fdw-test encoding 'LATIN1' lc_collate 'C' lc_ctype 'C' template template0;

\c mysql_fdw-test;

create extension mysql_fdw;
create server mysql-host foreign data wrapper mysql_fdw options (host '***.***.***.***', port '3306');
create user mapping for ******** server mysql-host options (username '********', password '********');

create schema test;

import foreign schema **** from server mysql into test;

create foreign table test.x2 (
     a int,
     b text
) server mysql-host options (dbname '****', table_name 'x');

select * from test.x2 limit 3;
select * from test.x where a > 5;

\det
SQL

# Clean-up
pg_ctl stop

Are you able to run (something like) the above (with the SQL adapted appropriately)? Are you able to give the actual SQL script that you were using for testing (with any sensitive details redacted)? Do you have a scripted build and test process that I could replicate on my end?

@jjthiessen jjthiessen force-pushed the import-foreign-schema branch from e25d2b8 to 7bcc63f Compare July 20, 2015 18:48
@ahsanhadi
Copy link

Ibrar has been able to reproduce the crash after applying your patch. The
crash is random in nature, the crash occurs because the patch is calling
this function directly from the mysql lib "mysql_num_rows". You need to
call this function as

    _mysql_stmt_bind_param = dlsym(mysql_dll_handle,

"mysql_stmt_bind_param");

Please see the comments for function mysql_load_library in mysql_fdw.c, it
explains why the function from mysql lib needs to called in this way.

Ibrar will commit a fix for this after applying your patch.

On Thu, Jul 9, 2015 at 12:52 AM, jjthiessen notifications@github.com
wrote:

@ahsanhadi https://github.com/ahsanhadi I tried to replicate the crash,
but I've been unable to do so thus far. However, there are potentially
significant differences between our environments:

  • I'm running MySQL 5.5 and I haven't setup a proper build env to test
    against different versions
  • I'm using the current HEAD of PostgreSQL's REL9_5_STABLE git branch
    (it sounded like you might be using the alpha1 tarball
    http://www.postgresql.org/ftp/source/v9.5alpha1/)
  • I'm merging my mysql_fdw:import-foreign-schema branch into an
    ephemeral fork of the official mysql_fdw:master to simulate the merge that
    would actually happen (I got the impression that you were manually applying
    patches or cherry-picking based on your earlier comments)).

I did encounter an issue with how I was quoting string literals (that
would cause imports to fail (without crashing) depending on execution
order). This particular quoting bug should be fixed in my most recent push.

While you'll have to amend the SQL test code and login parameters to match
your server, I've used something like the following to bootstrap, build,
and test (in reality, I generally use separate scripts for each component
and I don't do clean builds each time I test, but I have run the following
in my environment (with different SQL matching my MySQL setup), and it
executes without the crash that you mention):

#!/bin/bash

Directory structure setup

BASE=${1:-${HOME}/src/import-foreign-schema-test}
mkdir -p ${BASE}
&& cd ${BASE}
|| exit

PostgreSQL setup

if [[ ! -d "${BASE}/postgres" ]]; then
git clone https://github.com/postgres/postgres.git ${BASE}/postgres
fi

cd ${BASE}/postgres
&& git checkout REL9_5_STABLE
&& git pull
|| exit

HASH=$(git rev-parse HEAD)
HASHDIR=${BASE}/pg/${HASH}

export PGPORT=9595
export PGDATA=${HASHDIR}/data
export PATH=${HASHDIR}/bin:${PATH}

./configure --prefix=${HASHDIR}
&& make clean
&& make
&& make check install
|| exit

if [[ ! -d "${PGDATA}" ]]; then
initdb
fi

pg_ctl -l ${HASHDIR}/pg.log start

mysql_fdw setup

if [[ ! -d "${BASE}/mysql_fdw" ]]; then
git clone https://github.com/EnterpriseDB/mysql_fdw.git ${BASE}/mysql_fdw
git remote add jjthiessen https://github.com/jjthiessen/mysql_fdw.git
fi

cd ${BASE}/mysql_fdw
&& git checkout master
&& git pull
&& (git branch -D ephemeral; git checkout -b ephemeral)
&& git fetch --all
&& git merge jjthiessen/import-foreign-schema
&& (rm *.o *.so; make USE_PGXS=1)
&& make USE_PGXS=1 install
|| exit

mysql_fdw test cases

psql -d postgres <<SQL
\timing

show server_version_num;

drop database if exists mysql_fdw-test;
create database mysql_fdw-test encoding 'LATIN1' lc_collate 'C' lc_ctype 'C' template template0;

\c mysql_fdw-test;

create extension mysql_fdw;
create server mysql-host foreign data wrapper mysql_fdw options (host '_..**.', port '3306');
create user mapping for * server mysql-host options (username '
', password '
_*****');

create schema test;

import foreign schema **** from server mysql into test;

create foreign table test.x2 (
a int,
b text
) server mysql-host options (dbname '****', table_name 'x');

select * from test.x2 limit 3;
select * from test.x where a > 5;

\det
SQL

Clean-up

pg_ctl stop

Are you able to run (something like) the above (with the SQL adapted
appropriately)? Are you able to give the actual SQL script that you were
using for testing (with any sensitive details redacted)? Do you have a
scripted build and test process that I could replicate on my end?


Reply to this email directly or view it on GitHub
#62 (comment)
.

Ahsan Hadi
Snr Director Product Development
EnterpriseDB Corporation
The Enterprise Postgres Company

Phone: +92-51-8358874
Mobile: +92-333-5162114

Website: www.enterprisedb.com
EnterpriseDB Blog: http://blogs.enterprisedb.com/
Follow us on Twitter: http://www.twitter.com/enterprisedb

This e-mail message (and any attachment) is intended for the use of the
individual or entity to whom it is addressed. This message contains
information from EnterpriseDB Corporation that may be privileged,
confidential, or exempt from disclosure under applicable law. If you are
not the intended recipient or authorized to receive this for the intended
recipient, any use, dissemination, distribution, retention, archiving, or
copying of this communication is strictly prohibited. If you have received
this e-mail in error, please notify the sender immediately by reply e-mail
and delete this message.

@ibrarahmad
Copy link
Contributor

jjthiessen: Please rebase your patch, so I can merge the pull request.

On Thu, Jul 23, 2015 at 8:03 PM, Ahsan Hadi notifications@github.com
wrote:

Ibrar has been able to reproduce the crash after applying your patch. The
crash is random in nature, the crash occurs because the patch is calling
this function directly from the mysql lib "mysql_num_rows". You need to
call this function as

_mysql_stmt_bind_param = dlsym(mysql_dll_handle,
"mysql_stmt_bind_param");

Please see the comments for function mysql_load_library in mysql_fdw.c, it
explains why the function from mysql lib needs to called in this way.

Ibrar will commit a fix for this after applying your patch.

On Thu, Jul 9, 2015 at 12:52 AM, jjthiessen notifications@github.com
wrote:

@ahsanhadi https://github.com/ahsanhadi I tried to replicate the
crash,
but I've been unable to do so thus far. However, there are potentially
significant differences between our environments:

  • I'm running MySQL 5.5 and I haven't setup a proper build env to test
    against different versions
  • I'm using the current HEAD of PostgreSQL's REL9_5_STABLE git branch
    (it sounded like you might be using the alpha1 tarball
    http://www.postgresql.org/ftp/source/v9.5alpha1/)
  • I'm merging my mysql_fdw:import-foreign-schema branch into an

ephemeral fork of the official mysql_fdw:master to simulate the merge
that
would actually happen (I got the impression that you were manually
applying
patches or cherry-picking based on your earlier comments)).

I did encounter an issue with how I was quoting string literals (that
would cause imports to fail (without crashing) depending on execution
order). This particular quoting bug should be fixed in my most recent
push.

While you'll have to amend the SQL test code and login parameters to
match
your server, I've used something like the following to bootstrap, build,
and test (in reality, I generally use separate scripts for each component
and I don't do clean builds each time I test, but I have run the
following
in my environment (with different SQL matching my MySQL setup), and it
executes without the crash that you mention):

#!/bin/bash

Directory structure setup

BASE=${1:-${HOME}/src/import-foreign-schema-test}
mkdir -p ${BASE}
&& cd ${BASE}
|| exit

PostgreSQL setup

if [[ ! -d "${BASE}/postgres" ]]; then
git clone https://github.com/postgres/postgres.git ${BASE}/postgres
fi

cd ${BASE}/postgres
&& git checkout REL9_5_STABLE
&& git pull
|| exit

HASH=$(git rev-parse HEAD)
HASHDIR=${BASE}/pg/${HASH}

export PGPORT=9595
export PGDATA=${HASHDIR}/data
export PATH=${HASHDIR}/bin:${PATH}

./configure --prefix=${HASHDIR}
&& make clean
&& make
&& make check install
|| exit

if [[ ! -d "${PGDATA}" ]]; then
initdb
fi

pg_ctl -l ${HASHDIR}/pg.log start

mysql_fdw setup

if [[ ! -d "${BASE}/mysql_fdw" ]]; then
git clone https://github.com/EnterpriseDB/mysql_fdw.git
${BASE}/mysql_fdw
git remote add jjthiessen https://github.com/jjthiessen/mysql_fdw.git
fi

cd ${BASE}/mysql_fdw
&& git checkout master
&& git pull
&& (git branch -D ephemeral; git checkout -b ephemeral)
&& git fetch --all
&& git merge jjthiessen/import-foreign-schema
&& (rm *.o *.so; make USE_PGXS=1)
&& make USE_PGXS=1 install
|| exit

mysql_fdw test cases

psql -d postgres <<SQL
\timing

show server_version_num;

drop database if exists mysql_fdw-test;
create database mysql_fdw-test encoding 'LATIN1' lc_collate 'C' lc_ctype
'C' template template0;

\c mysql_fdw-test;

create extension mysql_fdw;
create server mysql-host foreign data wrapper mysql_fdw options (host
'_..**.', port '3306');
create user mapping for * server mysql-host options (username
'
', password '
_*****');

create schema test;

import foreign schema **** from server mysql into test;

create foreign table test.x2 (
a int,
b text
) server mysql-host options (dbname '****', table_name 'x');

select * from test.x2 limit 3;
select * from test.x where a > 5;

\det
SQL

Clean-up

pg_ctl stop

Are you able to run (something like) the above (with the SQL adapted
appropriately)? Are you able to give the actual SQL script that you were
using for testing (with any sensitive details redacted)? Do you have a
scripted build and test process that I could replicate on my end?


Reply to this email directly or view it on GitHub
<
https://github.com/EnterpriseDB/mysql_fdw/pull/62#issuecomment-119712883>
.

Ahsan Hadi
Snr Director Product Development
EnterpriseDB Corporation
The Enterprise Postgres Company

Phone: +92-51-8358874
Mobile: +92-333-5162114

Website: www.enterprisedb.com
EnterpriseDB Blog: http://blogs.enterprisedb.com/
Follow us on Twitter: http://www.twitter.com/enterprisedb

This e-mail message (and any attachment) is intended for the use of the
individual or entity to whom it is addressed. This message contains
information from EnterpriseDB Corporation that may be privileged,
confidential, or exempt from disclosure under applicable law. If you are
not the intended recipient or authorized to receive this for the intended
recipient, any use, dissemination, distribution, retention, archiving, or
copying of this communication is strictly prohibited. If you have received
this e-mail in error, please notify the sender immediately by reply e-mail
and delete this message.


Reply to this email directly or view it on GitHub
#62 (comment)
.

Ibrar Ahmed
EnterpriseDB http://www.enterprisedb.com

@jjthiessen jjthiessen force-pushed the import-foreign-schema branch from 7bcc63f to 4685f39 Compare July 23, 2015 23:38
@jjthiessen
Copy link
Contributor Author

I've amended and rebased my branch accordingly.

@ahsanhadi
Copy link

Can you please generate a pull request for this?

On Fri, Jul 24, 2015 at 5:17 AM, jjthiessen notifications@github.com
wrote:

I've amended and rebased my branch accordingly.


Reply to this email directly or view it on GitHub
#62 (comment)
.

Ahsan Hadi
Snr Director Product Development
EnterpriseDB Corporation
The Enterprise Postgres Company

Phone: +92-51-8358874
Mobile: +92-333-5162114

Website: www.enterprisedb.com
EnterpriseDB Blog: http://blogs.enterprisedb.com/
Follow us on Twitter: http://www.twitter.com/enterprisedb

This e-mail message (and any attachment) is intended for the use of the
individual or entity to whom it is addressed. This message contains
information from EnterpriseDB Corporation that may be privileged,
confidential, or exempt from disclosure under applicable law. If you are
not the intended recipient or authorized to receive this for the intended
recipient, any use, dissemination, distribution, retention, archiving, or
copying of this communication is strictly prohibited. If you have received
this e-mail in error, please notify the sender immediately by reply e-mail
and delete this message.

@jjthiessen
Copy link
Contributor Author

This (#62) is a PR for merging jjthiessen/mysql_fdw:import-foreign-schema into EnterpriseDB/mysql_fdw:master. Were you wanting a separate PR to a different branch?

@ahsanhadi
Copy link

It looks good to commit now...

On Fri, Jul 24, 2015 at 10:10 PM, jjthiessen notifications@github.com
wrote:

This (#62 #62) is a PR
for merging jjthiessen/mysql_fdw:import-foreign-schema into
EnterpriseDB/mysql_fdw:master. Were you wanting a separate PR to a
different branch?


Reply to this email directly or view it on GitHub
#62 (comment)
.

Ahsan Hadi
Snr Director Product Development
EnterpriseDB Corporation
The Enterprise Postgres Company

Phone: +92-51-8358874
Mobile: +92-333-5162114

Website: www.enterprisedb.com
EnterpriseDB Blog: http://blogs.enterprisedb.com/
Follow us on Twitter: http://www.twitter.com/enterprisedb

This e-mail message (and any attachment) is intended for the use of the
individual or entity to whom it is addressed. This message contains
information from EnterpriseDB Corporation that may be privileged,
confidential, or exempt from disclosure under applicable law. If you are
not the intended recipient or authorized to receive this for the intended
recipient, any use, dissemination, distribution, retention, archiving, or
copying of this communication is strictly prohibited. If you have received
this e-mail in error, please notify the sender immediately by reply e-mail
and delete this message.

ibrarahmad added a commit that referenced this pull request Jul 27, 2015
Initial `IMPORT FOREIGN SCHEMA` support
@ibrarahmad ibrarahmad merged commit bc24ec0 into EnterpriseDB:master Jul 27, 2015
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants