[sync] Divorce python sync test server chromiumsync.py from testserve…

…r.py

Various chrome test suites use the infrastructure in net::LocalTestServer and net/tools/testserver.py to create local test server instances against which to run automated tests. Sync tests use reference implementations of sync and xmpp servers, which build on the testserver infrastructure in net/. In the past, the sync testserver was small enough that it made sense for it to be a part of the testserver in net/. This, however, resulted in an unwanted dependency from net/ onto sync/, due to the sync proto modules needed to run a python sync server. Now that the sync testserver has grown considerably in scope, it is time to separate it out from net/ while reusing base testserver code, and eliminate the dependency from net/ onto sync/. This work also provides us with the opportunity to remove a whole bunch of dead pyauto sync test code in chrome/test/functional.

This patch does the following:

- Moves the native class LocalSyncTestServer from net/test/ to sync/test/.
- Moves chromiumsync{_test}.py and xmppserver{_test}.py from net/tools/testserver/ to sync/tools/testserver/.
- Removes all sync server specific code from net/.
- Adds a new sync_testserver.py runner script for the python sync test.
- Moves some base classes from testserver.py to testserver_base.py so they can be reused by sync_testserver.py.
- Audits all the python imports in testserver.py, testserver_base.py and sync_testserver.py to make sure there are no unnecessary / missing imports.
- Adds a new run_sync_testserver runner executable to launch a sync testserver.
- Removes a couple of static methods from LocalTestServer, that were being used by run_testserver, and refactors run_sync_testserver to use their non-static versions.
- Adds the ability to run both chromiumsync_test.py and xmppserver_test.py from run_sync_testserver.
- Fixes chromiumsync.py to undo / rectify some older changes that broke tests in chromiumsync_test.py.
- Adds a new test target called test_support_sync_testserver to sync.gyp.
- Removes the hacky dependency on sync_proto from net.gyp:net_test_support.
- Updates various gyp files across chrome to use the new sync testserver target.
- Audits dependencies of net_test_support, run_testserver, and the newly added targets.
- Fixes the android chrome testserver spawner script to account for the above changes.
- Removes all mentions of TYPE_SYNC from the pyauto TestServer shim.
- Deletes all (deprecated) pyauto sync tests. (They had all become broken over time, gotten disabled, and were all redundant due to their equivalent sync integration tests.)
- Removes all sync related pyauto hooks from TestingAutomationProvider, since they are no longer going to be used.
- Takes care of a TODO in safe_browser_testserver.py to remove an unnecessary code block.

Note: A majority of the bugs listed below are for individual pyauto sync tests. Deleting the sync pyauto test script fixes all these bugs in one fell swoop.

TBR=mattm@chromium.org
BUG=117559, 119403, 159731, 15016, 80329, 49378, 87642, 86949, 88679, 104227, 88593, 124913
TEST=run_testserver, run_sync_testserver, sync_integration_tests, sync_performance_tests. All chrome tests that use a testserver should continue to work.

Review URL: https://codereview.chromium.org/11971025

git-svn-id: svn://svn.chromium.org/chrome/trunk/src@177864 0039d316-1c4b-4281-b951-d872f2087c98

build/android/pylib/chrome_test_server_spawner.py

@@ -1,4 +1,4 @@
-# Copyright (c) 2012 The Chromium Authors. All rights reserved.
+# Copyright 2013 The Chromium Authors. All rights reserved.
 # Use of this source code is governed by a BSD-style license that can be
 # found in the LICENSE file.
 
@@ -24,18 +24,19 @@
 import ports
 
 
-# Path that are needed to import necessary modules when running testserver.py.
-os.environ['PYTHONPATH'] = os.environ.get('PYTHONPATH', '') + ':%s:%s:%s:%s' % (
-    os.path.join(constants.CHROME_DIR, 'third_party'),
-    os.path.join(constants.CHROME_DIR, 'third_party', 'tlslite'),
-    os.path.join(constants.CHROME_DIR, 'third_party', 'pyftpdlib', 'src'),
-    os.path.join(constants.CHROME_DIR, 'net', 'tools', 'testserver'))
+# Path that are needed to import necessary modules when launching a testserver.
+os.environ['PYTHONPATH'] = os.environ.get('PYTHONPATH', '') + (':%s:%s:%s:%s:%s'
+    % (os.path.join(constants.CHROME_DIR, 'third_party'),
+       os.path.join(constants.CHROME_DIR, 'third_party', 'tlslite'),
+       os.path.join(constants.CHROME_DIR, 'third_party', 'pyftpdlib', 'src'),
+       os.path.join(constants.CHROME_DIR, 'net', 'tools', 'testserver'),
+       os.path.join(constants.CHROME_DIR, 'sync', 'tools', 'testserver')))
 
 
 SERVER_TYPES = {
     'http': '',
     'ftp': '-f',
-    'sync': '--sync',
+    'sync': '',  # Sync uses its own script, and doesn't take a server type arg.
     'tcpecho': '--tcp-echo',
     'udpecho': '--udp-echo',
 }
@@ -207,8 +208,13 @@ def run(self):
     logging.info('Start running the thread!')
     self.wait_event.clear()
     self._GenerateCommandLineArguments()
-    command = [os.path.join(constants.CHROME_DIR, 'net', 'tools',
-                            'testserver', 'testserver.py')] + self.command_line
+    command = constants.CHROME_DIR
+    if self.arguments['server-type'] == 'sync':
+      command = [os.path.join(command, 'sync', 'tools', 'testserver',
+                              'sync_testserver.py')] + self.command_line
+    else:
+      command = [os.path.join(command, 'net', 'tools', 'testserver',
+                              'testserver.py')] + self.command_line
     logging.info('Running: %s', command)
     self.process = subprocess.Popen(command)
     if self.process:

chrome/browser/automation/testing_automation_provider.cc

@@ -1,4 +1,4 @@
-// Copyright 2012 The Chromium Authors. All rights reserved.
+// Copyright 2013 The Chromium Authors. All rights reserved.
 // Use of this source code is governed by a BSD-style license that can be
 // found in the LICENSE file.
 
@@ -84,8 +84,6 @@
 #include "chrome/browser/search_engines/template_url_service_factory.h"
 #include "chrome/browser/sessions/session_service_factory.h"
 #include "chrome/browser/sessions/session_tab_helper.h"
-#include "chrome/browser/sync/profile_sync_service.h"
-#include "chrome/browser/sync/profile_sync_service_factory.h"
 #include "chrome/browser/ui/app_modal_dialogs/app_modal_dialog.h"
 #include "chrome/browser/ui/app_modal_dialogs/app_modal_dialog_queue.h"
 #include "chrome/browser/ui/app_modal_dialogs/javascript_app_modal_dialog.h"
@@ -1910,19 +1908,6 @@ void TestingAutomationProvider::BuildJSONHandlerMaps() {
   browser_handler_map_["WaitForNotificationCount"] =
       &TestingAutomationProvider::WaitForNotificationCount;
 
-  browser_handler_map_["SignInToSync"] =
-      &TestingAutomationProvider::SignInToSync;
-  browser_handler_map_["GetSyncInfo"] =
-      &TestingAutomationProvider::GetSyncInfo;
-  browser_handler_map_["AwaitFullSyncCompletion"] =
-      &TestingAutomationProvider::AwaitFullSyncCompletion;
-  browser_handler_map_["AwaitSyncRestart"] =
-      &TestingAutomationProvider::AwaitSyncRestart;
-  browser_handler_map_["EnableSyncForDatatypes"] =
-      &TestingAutomationProvider::EnableSyncForDatatypes;
-  browser_handler_map_["DisableSyncForDatatypes"] =
-      &TestingAutomationProvider::DisableSyncForDatatypes;
-
   browser_handler_map_["GetNTPInfo"] =
       &TestingAutomationProvider::GetNTPInfo;
   browser_handler_map_["RemoveNTPMostVisitedThumbnail"] =
@@ -4178,237 +4163,6 @@ void TestingAutomationProvider::AppendSwitchASCIIToCommandLine(
   reply.SendSuccess(NULL);
 }
 
-// Sample json output: { "success": true }
-void TestingAutomationProvider::SignInToSync(Browser* browser,
-                                             DictionaryValue* args,
-                                             IPC::Message* reply_message) {
-  AutomationJSONReply reply(this, reply_message);
-  std::string username;
-  std::string password;
-  if (!args->GetString("username", &username) ||
-      !args->GetString("password", &password)) {
-      reply.SendError("Invalid or missing args");
-      return;
-  }
-  if (sync_waiter_.get() == NULL) {
-    sync_waiter_.reset(new ProfileSyncServiceHarness(
-        browser->profile(), username, password));
-  } else {
-    sync_waiter_->SetCredentials(username, password);
-  }
-  if (sync_waiter_->SetupSync()) {
-    scoped_ptr<DictionaryValue> return_value(new DictionaryValue);
-    return_value->SetBoolean("success", true);
-    reply.SendSuccess(return_value.get());
-  } else {
-    reply.SendError("Signing in to sync was unsuccessful");
-  }
-}
-
-// Sample json output:
-// {u'summary': u'SYNC DISABLED'}
-//
-// { u'last synced': u'Just now',
-//   u'sync url': u'clients4.google.com',
-//   u'updates received': 42,
-//   u'synced datatypes': [ u'Bookmarks',
-//                          u'Preferences',
-//                          u'Passwords',
-//                          u'Autofill',
-//                          u'Themes',
-//                          u'Extensions',
-//                          u'Apps']}
-void TestingAutomationProvider::GetSyncInfo(Browser* browser,
-                                            DictionaryValue* args,
-                                            IPC::Message* reply_message) {
-  AutomationJSONReply reply(this, reply_message);
-  DictionaryValue* sync_info = new DictionaryValue;
-  scoped_ptr<DictionaryValue> return_value(new DictionaryValue);
-  if (sync_waiter_.get() == NULL) {
-    sync_waiter_.reset(
-        ProfileSyncServiceHarness::CreateAndAttach(browser->profile()));
-  }
-  if (!sync_waiter_->IsSyncAlreadySetup()) {
-    sync_info->SetString("summary", "SYNC DISABLED");
-  } else {
-    ProfileSyncService* service = sync_waiter_->service();
-    ProfileSyncService::Status status = sync_waiter_->GetStatus();
-    sync_info->SetString("sync url", service->sync_service_url().host());
-    sync_info->SetString("last synced", service->GetLastSyncedTimeString());
-    sync_info->SetInteger("updates received", status.updates_received);
-    ListValue* synced_datatype_list = new ListValue;
-    const syncer::ModelTypeSet synced_datatypes =
-        service->GetPreferredDataTypes();
-    for (syncer::ModelTypeSet::Iterator it = synced_datatypes.First();
-         it.Good(); it.Inc()) {
-      synced_datatype_list->Append(
-          new StringValue(syncer::ModelTypeToString(it.Get())));
-    }
-    sync_info->Set("synced datatypes", synced_datatype_list);
-  }
-  return_value->Set("sync_info", sync_info);
-  reply.SendSuccess(return_value.get());
-}
-
-// Sample json output: { "success": true }
-void TestingAutomationProvider::AwaitFullSyncCompletion(
-    Browser* browser,
-    DictionaryValue* args,
-    IPC::Message* reply_message) {
-  AutomationJSONReply reply(this, reply_message);
-  if (sync_waiter_.get() == NULL) {
-    sync_waiter_.reset(
-        ProfileSyncServiceHarness::CreateAndAttach(browser->profile()));
-  }
-  if (!sync_waiter_->IsSyncAlreadySetup()) {
-    reply.SendError("Not signed in to sync");
-    return;
-  }
-  // Ensure that the profile sync service and sync backend host are initialized
-  // before waiting for sync cycle completion. In cases where the browser is
-  // restarted with sync enabled, these operations may still be in flight.
-  if (ProfileSyncServiceFactory::GetInstance()->GetForProfile(
-          browser->profile()) == NULL) {
-    reply.SendError("ProfileSyncService initialization failed.");
-    return;
-  }
-  if (!sync_waiter_->service()->sync_initialized() &&
-      !sync_waiter_->AwaitBackendInitialized()) {
-    reply.SendError("Sync backend host initialization failed.");
-    return;
-  }
-  if (!sync_waiter_->AwaitFullSyncCompletion("Waiting for sync cycle")) {
-    reply.SendError("Sync cycle did not complete.");
-    return;
-  }
-  scoped_ptr<DictionaryValue> return_value(new DictionaryValue);
-  return_value->SetBoolean("success", true);
-  reply.SendSuccess(return_value.get());
-}
-
-// Sample json output: { "success": true }
-void TestingAutomationProvider::AwaitSyncRestart(
-    Browser* browser,
-    DictionaryValue* args,
-    IPC::Message* reply_message) {
-  AutomationJSONReply reply(this, reply_message);
-  if (sync_waiter_.get() == NULL) {
-    sync_waiter_.reset(
-        ProfileSyncServiceHarness::CreateAndAttach(browser->profile()));
-  }
-  if (!sync_waiter_->IsSyncAlreadySetup()) {
-    reply.SendError("Not signed in to sync");
-    return;
-  }
-  if (ProfileSyncServiceFactory::GetInstance()->GetForProfile(
-          browser->profile()) == NULL) {
-    reply.SendError("ProfileSyncService initialization failed.");
-    return;
-  }
-  if (!sync_waiter_->AwaitSyncRestart()) {
-    reply.SendError("Sync did not successfully restart.");
-    return;
-  }
-  scoped_ptr<DictionaryValue> return_value(new DictionaryValue);
-  return_value->SetBoolean("success", true);
-  reply.SendSuccess(return_value.get());
-}
-
-// Refer to EnableSyncForDatatypes() in chrome/test/pyautolib/pyauto.py for
-// sample json input. Sample json output: { "success": true }
-void TestingAutomationProvider::EnableSyncForDatatypes(
-    Browser* browser,
-    DictionaryValue* args,
-    IPC::Message* reply_message) {
-  AutomationJSONReply reply(this, reply_message);
-  if (sync_waiter_.get() == NULL) {
-    sync_waiter_.reset(
-        ProfileSyncServiceHarness::CreateAndAttach(browser->profile()));
-  }
-  if (!sync_waiter_->IsSyncAlreadySetup()) {
-    reply.SendError("Not signed in to sync");
-    return;
-  }
-  ListValue* datatypes = NULL;
-  if (!args->GetList("datatypes", &datatypes)) {
-    reply.SendError("Invalid or missing args");
-    return;
-  }
-  std::string first_datatype;
-  datatypes->GetString(0, &first_datatype);
-  if (first_datatype == "All") {
-    sync_waiter_->EnableSyncForAllDatatypes();
-  } else {
-    int num_datatypes = datatypes->GetSize();
-    for (int i = 0; i < num_datatypes; ++i) {
-      std::string datatype_string;
-      datatypes->GetString(i, &datatype_string);
-      syncer::ModelType datatype =
-          syncer::ModelTypeFromString(datatype_string);
-      if (datatype == syncer::UNSPECIFIED) {
-        AutomationJSONReply(this, reply_message).SendError(StringPrintf(
-            "Invalid datatype string: %s.", datatype_string.c_str()));
-        return;
-      }
-      sync_waiter_->EnableSyncForDatatype(datatype);
-      sync_waiter_->AwaitFullSyncCompletion(StringPrintf(
-          "Enabling datatype: %s", datatype_string.c_str()));
-    }
-  }
-  scoped_ptr<DictionaryValue> return_value(new DictionaryValue);
-  return_value->SetBoolean("success", true);
-  reply.SendSuccess(return_value.get());
-}
-
-// Refer to DisableSyncForDatatypes() in chrome/test/pyautolib/pyauto.py for
-// sample json input. Sample json output: { "success": true }
-void TestingAutomationProvider::DisableSyncForDatatypes(
-    Browser* browser,
-    DictionaryValue* args,
-    IPC::Message* reply_message) {
-  AutomationJSONReply reply(this, reply_message);
-  if (sync_waiter_.get() == NULL) {
-    sync_waiter_.reset(
-        ProfileSyncServiceHarness::CreateAndAttach(browser->profile()));
-  }
-  if (!sync_waiter_->IsSyncAlreadySetup()) {
-    reply.SendError("Not signed in to sync");
-    return;
-  }
-  ListValue* datatypes = NULL;
-  if (!args->GetList("datatypes", &datatypes)) {
-    reply.SendError("Invalid or missing args");
-    return;
-  }
-  std::string first_datatype;
-  if (!datatypes->GetString(0, &first_datatype)) {
-    reply.SendError("Invalid or missing string");
-    return;
-  }
-  if (first_datatype == "All") {
-    sync_waiter_->DisableSyncForAllDatatypes();
-  } else {
-    int num_datatypes = datatypes->GetSize();
-    for (int i = 0; i < num_datatypes; i++) {
-      std::string datatype_string;
-      datatypes->GetString(i, &datatype_string);
-      syncer::ModelType datatype =
-          syncer::ModelTypeFromString(datatype_string);
-      if (datatype == syncer::UNSPECIFIED) {
-        AutomationJSONReply(this, reply_message).SendError(StringPrintf(
-            "Invalid datatype string: %s.", datatype_string.c_str()));
-        return;
-      }
-      sync_waiter_->DisableSyncForDatatype(datatype);
-      sync_waiter_->AwaitFullSyncCompletion(StringPrintf(
-          "Disabling datatype: %s", datatype_string.c_str()));
-    }
-    scoped_ptr<DictionaryValue> return_value(new DictionaryValue);
-    return_value->SetBoolean("success", true);
-    reply.SendSuccess(return_value.get());
-  }
-}
-
 // Refer to GetAllNotifications() in chrome/test/pyautolib/pyauto.py for
 // sample json input/output.
 void TestingAutomationProvider::GetAllNotifications(

chrome/browser/automation/testing_automation_provider.h

@@ -1,4 +1,4 @@
-// Copyright (c) 2012 The Chromium Authors. All rights reserved.
+// Copyright 2013 The Chromium Authors. All rights reserved.
 // Use of this source code is governed by a BSD-style license that can be
 // found in the LICENSE file.
 
@@ -18,7 +18,6 @@
 #include "chrome/browser/automation/automation_provider_json.h"
 #include "chrome/browser/history/history.h"
 #include "chrome/browser/importer/importer_list_observer.h"
-#include "chrome/browser/sync/profile_sync_service_harness.h"
 #include "chrome/browser/ui/browser_list_observer.h"
 #include "content/public/browser/notification_registrar.h"
 #include "content/public/common/page_type.h"
@@ -565,42 +564,6 @@ class TestingAutomationProvider : public AutomationProvider,
   void IsFindInPageVisible(base::DictionaryValue* args,
                            IPC::Message* reply_message);
 
-  // Signs in to sync using the given username and password.
-  // Uses the JSON interface for input/output.
-  void SignInToSync(Browser* browser,
-                    base::DictionaryValue* args,
-                    IPC::Message* reply_message);
-
-  // Returns info about sync.
-  // Uses the JSON interface for input/output.
-  void GetSyncInfo(Browser* browser,
-                   base::DictionaryValue* args,
-                   IPC::Message* reply_message);
-
-  // Waits for the ongoing sync cycle to complete.
-  // Uses the JSON interface for input/output.
-  void AwaitFullSyncCompletion(Browser* browser,
-                               base::DictionaryValue* args,
-                               IPC::Message* reply_message);
-
-  // Waits for sync to reinitialize (for example, after a browser restart).
-  // Uses the JSON interface for input/output.
-  void AwaitSyncRestart(Browser* browser,
-                        base::DictionaryValue* args,
-                        IPC::Message* reply_message);
-
-  // Enables sync for one or more sync datatypes.
-  // Uses the JSON interface for input/output.
-  void EnableSyncForDatatypes(Browser* browser,
-                              base::DictionaryValue* args,
-                              IPC::Message* reply_message);
-
-  // Disables sync for one or more sync datatypes.
-  // Uses the JSON interface for input/output.
-  void DisableSyncForDatatypes(Browser* browser,
-                               base::DictionaryValue* args,
-                               IPC::Message* reply_message);
-
   // Get ordered list of all active and queued HTML5 notifications.
   // Uses the JSON interface for input/output.
   void GetAllNotifications(Browser* browser,
@@ -1559,9 +1522,6 @@ class TestingAutomationProvider : public AutomationProvider,
   std::map<std::string, JsonHandler> handler_map_;
   std::map<std::string, BrowserJsonHandler> browser_handler_map_;
 
-  // Used to wait on various browser sync events.
-  scoped_ptr<ProfileSyncServiceHarness> sync_waiter_;
-
   content::NotificationRegistrar registrar_;
 
   // Used to enumerate browser profiles.