The problem is actually that the function smtpmail-open-stream uses a
macro `with-boundp' that is not defined in 21.4.22, but only in 21.5
(http://membled.com/lib/xemacs-21.5-b27/lisp/bytecomp-runtime.el)

Others have run into this problem:
http://groups.google.com/group/gnu.emacs.help/browse_thread/thread/2538363186504a06/ba2a2effcce72898

Therefore the OS process is never created and an internal network stream
is created instead.

There is a workaround:
1) Put the definition of `with-boundp' in the .gnus file.
2) Disable the bytecompiled smtpmail or redefnie `smtpmail-open-stream'
in your .gnus after it has been loaded from you OS distibuted version.

That will fix it.

So basically put this in top of your .gnus:
-------------------------
;;A fix for xemacs mail-lib
(defmacro with-boundp (variables &rest body)
  "Evaluate BODY, but do not issue bytecomp warnings about VARIABLES
undefined.
VARIABLES can be a symbol or a list of symbols and must be quoted.  When
compiling this file, the warnings `reference to free variable VARIABLE' and
`assignment to free variable VARIABLE' will not occur anywhere in BODY, for
any of the listed variables.  This is a clean way to avoid such warnings.

See also `if-boundp', `when-boundp', and `and-boundp' (ways to
conditionalize on a variable being bound and avoid warnings),
`declare-boundp' (issue a variable call without warnings), and
`globally-declare-boundp' (avoid warnings throughout a file about a
variable)."
  (setq variables (eval variables))
  (unless (consp variables)
    (setq variables (list variables)))
  `(progn
     (declare (special ,@variables))
     ,@body))
(require 'smtpmail)
(defun smtpmail-open-stream (process-buffer host port)
  (let ((cred (smtpmail-find-credentials
	       smtpmail-starttls-credentials host port)))
    (if (null (and cred (condition-case ()
			    ;; XEmacs change
			    (with-boundp '(starttls-use-gnutls
					   starttls-gnutls-program
					   starttls-program)
			      (require 'starttls)
			      (call-process (if starttls-use-gnutls
						starttls-gnutls-program
					      starttls-program)))
			  (error nil))))
	;; The normal case.
	(open-network-stream "SMTP" process-buffer host port)
      (let* ((cred-key (smtpmail-cred-key cred))
	     (cred-cert (smtpmail-cred-cert cred))
	     (starttls-extra-args
	      (append
	       starttls-extra-args
	       (when (and (stringp cred-key) (stringp cred-cert)
			  (file-regular-p
			   (setq cred-key (expand-file-name cred-key)))
			  (file-regular-p
			   (setq cred-cert (expand-file-name cred-cert))))
		 (list "--key-file" cred-key "--cert-file" cred-cert))))
	     (starttls-extra-arguments
	      (append
	       starttls-extra-arguments
	       (when (and (stringp cred-key) (stringp cred-cert)
			  (file-regular-p
			   (setq cred-key (expand-file-name cred-key)))
			  (file-regular-p
			   (setq cred-cert (expand-file-name cred-cert))))
		 (list "--x509keyfile" cred-key "--x509certfile" cred-cert)))))
	(starttls-open-stream "SMTP" process-buffer host port)))))
------------------------------------

Jarl