- Initial simple msg test case.

- Update TODO, stripping out all the redundant ole stuff.
- Fix property set guids to use the new Ole::Types::Clsid type.
- Add block form Msg.open
- Fix file requires for running tests individually.
- Start to flesh out move to mapi
- Update RangesIO subclasses for changes in ruby-ole


git-svn-id: https://ruby-msg.googlecode.com/svn/trunk@113 c30d66de-b626-0410-988f-81f6512a6d81

Author: aquasync

Rakefile

@@ -17,7 +17,7 @@ task :default => [:test]
 
 Rake::TestTask.new(:test) do |t|
 	t.test_files = FileList["test/test_*.rb"]
-	t.warning = true
+	t.warning = false
 	t.verbose = true
 end
 
@@ -51,7 +51,7 @@ spec = Gem::Specification.new do |s|
 
 	s.autorequire = 'msg'
 
-	s.add_dependency 'ruby-ole', '>=1.2.1'
+	s.add_dependency 'ruby-ole', '>=1.2.3'
 end
 
 Rake::GemPackageTask.new(spec) do |p|

TODO

@@ -1,48 +1,10 @@
-= Ole
-
-* support for shrinking the bats. sbat and bbat can grow to accomodate more data,
-  but they can't shrink yet. at least they should be cropped so that all trailing
-  AVAIL are removed at save time, then re-padded to block boundary.
-  (solved by truncated_table ???)
-* similarly, at close/save time, we should truncate the parent io file.
-* wipe-out with 0's on truncation time not yet done (save on junk).
-* handle including bat information in the bat itself (increasingly important)
-  (partial support)
-* how to handle a small block file getting bigger than threshold, or big block
-  file getting smaller than it?? when you start writing to an empty file, it
-  should be sbat by rights, but then it gets past a certain point, and we have
-  to transparently map that 4096 bytes to a bbat file and keep going??? seems
-  incredibly messy, and will cause a lot of churn in the allocation tables.
-  neater solution???
-  maybe handle it at close time somehow? all new chains are bbat, and then get
-  migrated to sbat at close time? might be less expensive.
-  "solution" for now: buffering. IO.copy buffer is 4096. increase this, to 
-  8192, then its bigger than default migration threshold (4096), so first
-  write to unallocated file moves it directly to bbat with no sbat involvement.
-  if you do smaller writes, you will end up claiming a lot of sbat blocks,
-  and move to bbat later. buffering would help too, but i'll rely on IO.copy
-  for now.
-  (bat migration implemented)
-* when all the above is implemented, we'll allow cheaper edits that our current
-  complete-rewrite approach, but they'll be messier. so, support a repack function.
-  it opens a temporary file, completely copies @io to it, opens it as a Storage
-  object, truncates @io, resets things like initialize would (clean header etc).
-  then it does a Dirent#copy on the 2 roots to provide a clean version.
-  thus:
-  
-  Ole::Storage.open 'myfile.doc', 'r+', &:repack
-  
-  would be a simple ole file cleaner.
-
-* then test & cleanup.
-
 = Newer Msg
 
 * fix nameid handling for sub Properties objects. needs to inherit top-level @nameid.
 
 * better handling for "Untitled Attachments", or attachments with no filename at all.
   maybe better handling in general for attached messages. maybe re-write filename with
-	subject.
+  subject.
 
 * do the above 2, then go for a new release.
   for this release, i want it to be pretty robust, and have better packaging.
@@ -52,16 +14,16 @@
   submit to the usual gem repositories etc, announce project as usable.
 
 * better handling for other types of embedded ole attachments. try to recognise .wmf to
-	start with. etc. 2 goals, form .eml, and for .rtf output.
+  start with. etc. 2 goals, form .eml, and for .rtf output.
 
 * rtf to text support, initially. just simple strip all the crap out approach. maybe html
-	later on. sometimes the only body is rtf, which is problematic. is rtf an allowed body
-	type?
+  later on. sometimes the only body is rtf, which is problematic. is rtf an allowed body
+  type?
 
 * better encoding support (i'm thinking, check some things like m-dash in the non-wide
-	string type. is it windows codepage?. how to know code page in general? change
-	ENCODER[] to convert these to utf8 i think). I'm trying to just move everything to utf8.
-	then i need to make sure the mime side of that holds up.
+  string type. is it windows codepage?. how to know code page in general? change
+  ENCODER[] to convert these to utf8 i think). I'm trying to just move everything to utf8.
+  then i need to make sure the mime side of that holds up.
 
 * certain parsing still not correct, especially small properties directory, 
   things like bools, multiavlue shorts etc.

lib/mapi.rb

@@ -1,5 +1,47 @@
 require 'msg'
 
+=begin
+
+Mapi::PropertyStore
+Mapi::Item
+Mapi::Attachment
+Mapi::Recipient
+
+Pst::PropertyStore < Mapi::PropertyStore
+Msg::PropertyStore < Mapi::PropertyStore
+
+# simple hash like api. can be lazily loaded. however if you
+# use something like #keys, or #values, then you'd probably
+# just cache things. maybe this would be the raw interface, 
+# which is wrapped by something which provides named access.
+class Mapi::PropertyStore
+	def initialize symbolic=true
+		@symbolic = symbolic
+	end
+
+	def raw
+		# gives access to the real property store object. the top level
+		# wrapped one uses symbolic naming. 
+		self.class.new false
+	end
+
+	def each symbolic=@symbolic
+		# iterate through the key value pairs. keys are
+		# Mapi::PropertyStore::Key instances, which allows for named
+		# properties. values are of the appropriate class (which is
+		# what?). 
+	end
+
+	def []
+		# get a value based on a key
+	end
+
+	def to_h symbolic=@symbolic
+	end
+end
+
+=end
+
 module Mapi
 	module Types
 		#

lib/mime.rb

@@ -1,3 +1,7 @@
+# need RecursivelyEnumerable
+require 'rubygems'
+require 'ole/support'
+
 #
 # = Introduction
 #

lib/msg.rb

@@ -1,7 +1,3 @@
-#! /usr/bin/ruby
-
-$: << File.dirname(__FILE__)
-
 require 'yaml'
 require 'base64'
 
@@ -34,10 +30,15 @@ class Msg
 	# Alternate constructor, to create an +Msg+ directly from +arg+ and +mode+, passed
 	# directly to Ole::Storage (ie either filename or seekable IO object).
 	def self.open arg, mode=nil
-		msg = Msg.new Ole::Storage.open(arg, mode).root
+		msg = new Ole::Storage.open(arg, mode).root
 		# we will close the ole when we are #closed
 		msg.close_parent = true
-		msg
+		if block_given?
+			begin   yield msg
+			ensure; msg.close
+			end
+		else msg
+		end
 	end
 
 	# Create an Msg from +root+, an <tt>Ole::Storage::Dirent</tt> object
@@ -162,7 +163,7 @@ def populate_headers
 			time = nil unless Date === time
 			# can employ other methods for getting a time. heres one in a similar vein to msgconvert.pl,
 			# ie taking the time from an ole object
-			time ||= @root.ole.dirents.map(&:time).compact.sort.last
+			time ||= @root.ole.dirents.map { |dirent| dirent.modify_time || dirent.create_time }.compact.sort.last
 
 			# now convert and store
 			# this is a little funky. not sure about time zone stuff either?
@@ -510,16 +511,3 @@ def inspect
 	end
 end
 
-if $0 == __FILE__
-	quiet = if ARGV[0] == '-q'
-		ARGV.shift
-		true
-	end
-	# just shut up and convert a message to eml
-	Msg::Log.level = Logger::WARN
-	Msg::Log.level = Logger::FATAL if quiet
-	msg = Msg.open ARGV[0]
-	puts msg.to_mime.to_s
-	msg.close
-end
-

lib/msg/properties.rb

@@ -107,30 +107,46 @@ class Properties
 			:default => proc { |obj| obj.open }
 		}
 
-		# these won't be strings for much longer.
-		# maybe later, the Key#inspect could automatically show symbolic guid names if they
-		# are part of this builtin list.
-		# FIXME. hey, nice that my fake string is the same length though :)
-		PS_MAPI =             '{not-really-sure-what-this-should-say}'
-		PS_PUBLIC_STRINGS =   '{00020329-0000-0000-c000-000000000046}'
-		# string properties in this namespace automatically get added to the internet headers
-		PS_INTERNET_HEADERS = '{00020386-0000-0000-c000-000000000046}'
-		# theres are bunch of outlook ones i think
-		# http://blogs.msdn.com/stephen_griffin/archive/2006/05/10/outlook-2007-beta-documentation-notification-based-indexing-support.aspx
-		# IPM.Appointment
-		PSETID_Appointment =  '{00062002-0000-0000-c000-000000000046}'
-		# IPM.Task
-		PSETID_Task =         '{00062003-0000-0000-c000-000000000046}'
-		# used for IPM.Contact
-		PSETID_Address =      '{00062004-0000-0000-c000-000000000046}'
-		PSETID_Common =       '{00062008-0000-0000-c000-000000000046}'
-		# didn't find a source for this name. it is for IPM.StickyNote
-		PSETID_Note =         '{0006200e-0000-0000-c000-000000000046}'
-		# for IPM.Activity. also called the journal?
-		PSETID_Log =          '{0006200a-0000-0000-c000-000000000046}'
-
 		SUBSTG_RX = /__substg1\.0_([0-9A-F]{4})([0-9A-F]{4})(?:-([0-9A-F]{8}))?/
 
+		# the property set guid constants
+		# these guids are all defined with the macro DEFINE_OLEGUID in mapiguid.h.
+		# see http://doc.ddart.net/msdn/header/include/mapiguid.h.html
+		oleguid = proc do |prefix|
+			Ole::Types::Clsid.parse "{#{prefix}-0000-0000-c000-000000000046}"
+		end
+
+		NAMES = {
+			oleguid['00020328'] => 'PS_MAPI',
+			oleguid['00020329'] => 'PS_PUBLIC_STRINGS',
+			oleguid['00020380'] => 'PS_ROUTING_EMAIL_ADDRESSES',
+			oleguid['00020381'] => 'PS_ROUTING_ADDRTYPE',
+			oleguid['00020382'] => 'PS_ROUTING_DISPLAY_NAME',
+			oleguid['00020383'] => 'PS_ROUTING_ENTRYID',
+			oleguid['00020384'] => 'PS_ROUTING_SEARCH_KEY',
+			# string properties in this namespace automatically get added to the internet headers
+			oleguid['00020386'] => 'PS_INTERNET_HEADERS',
+			# theres are bunch of outlook ones i think
+			# http://blogs.msdn.com/stephen_griffin/archive/2006/05/10/outlook-2007-beta-documentation-notification-based-indexing-support.aspx
+			# IPM.Appointment
+			oleguid['00062002'] => 'PSETID_Appointment',
+			# IPM.Task
+			oleguid['00062003'] => 'PSETID_Task',
+			# used for IPM.Contact
+			oleguid['00062004'] => 'PSETID_Address',
+			oleguid['00062008'] => 'PSETID_Common',
+			# didn't find a source for this name. it is for IPM.StickyNote
+			oleguid['0006200e'] => 'PSETID_Note',
+			# for IPM.Activity. also called the journal?
+			oleguid['0006200a'] => 'PSETID_Log',
+		}
+
+		module Constants
+			NAMES.each { |guid, name| const_set name, guid }
+		end
+
+		include Constants
+
 		# access the underlying raw property hash
 		attr_reader :raw
 		# unused (non-property) objects after parsing an +Dirent+.
@@ -453,6 +469,8 @@ def body_html
 		#
 		# Also contains the code that maps keys to symbolic names.
 		class Key
+			include Constants
+
 			attr_reader :code, :guid
 			def initialize code, guid=PS_MAPI
 				@code, @guid = code, guid
@@ -502,17 +520,21 @@ def == other
 			alias eql? :==
 
 			def inspect
+				# maybe the way to do this, would be to be able to register guids
+				# in a global lookup, which are used by Clsid#inspect itself, to
+				# provide symbolic names...
+				guid_str = NAMES[guid] || "{#{guid.format}}"
 				if Integer === code
 					hex = '0x%04x' % code
 					if guid == PS_MAPI
 						# just display as plain hex number
 						hex
 					else
-						"#<Key #{guid}/#{hex}>"
+						"#<Key #{guid_str}/#{hex}>"
 					end
 				else
 					# display full guid and code
-					"#<Key #{guid}/#{code.inspect}>"
+					"#<Key #{guid_str}/#{code.inspect}>"
 				end
 			end
 		end

lib/pst.rb

@@ -7,12 +7,14 @@
 #
 # = TODO
 # 
-# 0. massive refactoring and cleaning up, now that the main stuff is supported.
-# 1. xattribs
-# 2. solve recipient table problem (test4).
-# 3. generalise the Mapi stuff better
-# 4. refactor index load
-# 5. msg serialization?
+# 1. solve recipient table problem (test4).
+#    this is done. turns out it was due to id2 clashes. find better solution
+# 2. check parse consistency. an initial conversion of a 30M file to pst, shows
+#    a number of messages conveting badly. compare with libpst too.
+# 3. xattribs
+# 4. generalise the Mapi stuff better
+# 5. refactor index load
+# 6. msg serialization?
 #
 
 =begin
@@ -238,8 +240,9 @@ def self.encrypt decrypted
 	end
 
 	class RangesIOEncryptable < RangesIO
-		def initialize io, ranges, opts={}
-			@decrypt = !!opts[:decrypt]
+		def initialize io, mode='r', params={}
+			mode, params = 'r', mode if Hash === mode
+			@decrypt = !!params[:decrypt]
 			super
 		end
 
@@ -880,7 +883,7 @@ def initialize pst, idx_head
 			decrypt = decrypts.first
 			# convert idxs to ranges
 			ranges = @idxs.map { |idx| [idx.offset, idx.size] }
-			super pst.io, ranges, :decrypt => decrypt
+			super pst.io, :ranges => ranges, :decrypt => decrypt
 		end
 	end
 

lib/rtf.rb

@@ -1,5 +1,3 @@
-#! /usr/bin/ruby -w
-
 require 'stringio'
 
 # this file is pretty crap, its just to ensure there is always something readable if
@@ -109,10 +107,3 @@ def self.rtf2text str, format=:text
 	end
 end
 
-if $0 == __FILE__
-	#str = File.read('test.rtf')
-	str = YAML.load(open('rtfs.yaml'))[2]
-	#puts str
-	puts text
-end
-

test/test_Blammo.msg

@@ -14,9 +14,11 @@ def test_parsing_no_multipart
 		assert_equal 'Body text.', mime.body
 		assert_equal false, mime.multipart?
 		assert_equal nil, mime.parts
-		# we get round trip conversion. this is mostly fluke, as orderedhash hasn't been
-		# added yet
 		assert_equal "Header1: Value1\r\nHeader2: Value2\r\n\r\nBody text.", mime.to_s
 	end
+	
+	def test_boundaries
+		assert_match(/^----_=_NextPart_001_/, Mime.make_boundary(1))
+	end
 end