This topic is locked

Sending bulk e-mails...

3/24/2007 6:32:35 AM
PHPRunner General questions
M
mmponline author

With the great help of support, we developed a script to send e-mails to registered users of certain info added to a table (Eg. adding a new topic)
We do have some problems, however. The script sends e-mails if the number of users are limited. As soon as it becomes more, the script doesn't send at all. I realise this is not a PHPRunner limitation, rather something to do with mail sending limitations.
Anyone with advice onthe following:

  1. Is there a limit and what is it?
  2. Is there another wa to send, as I know bulk sending works over the net via newsletter programs (php generated)
  3. Any help with scripts of event to achieve this?
    Here is the script we currently use:

function BeforeAdd(&$values)

{
// Parameters:

// $values - Array object.

// Each field on the Add form represented as 'Field name'-'Field value' pair
//********** Send email with new data ************
global $conn;

$all_emails="";

$mailfrom1 = "admin@adminemailaddress.co.za";

$strSQL = "select Email from Login where Available='Yes please'";

$rs = db_query($strSQL,$conn);

while ($data = db_fetch_array($rs))

$all_emails .= $data["Email"].", ";

$all_emails = substr($all_emails,0,strlen($all_emails)-2);

$message="The following rentals are available. Please book at our office. For more info on rental availability,login to your account on our website: www.verlorenkloof.co.za\r\n";;
$subject="Rental Availability:";
$message.= "From : ".$values["c_from"]."\r";

$message.= "To : ".$values["c_to"]."\r";

$message.= "Croft Number : ".$values["c_croftid"]."\r";

$message.= "Rate : ".$values["c_rate"]."\r";

$message.= "Beds : ".$values["c_beds"]."\r";

$message.= "Contact and Details :\r ".$values["c_admin"]."\r";
mail($all_emails, $subject, $message, 'From: '.$mailfrom1);
return true;
// return true if you like to proceed with adding new record

// return false in other case
}


Thanks for any help to get this working...

D
Dale 3/24/2007

Im no expert but your code seems fine upto and including the line below.

$all_emails = substr($all_emails,0,strlen($all_emails)-2);
After that, I would try to get the snippet below into your code. I bulk email customers and have not had any problems yet.

My customer batches out anywhere from 10 to 600 emails on a normal run with the largest email batch of 1600+ and no problems. I investigated this quite some time ago. I even tried using some of the bulk email php code utilities but had too many issues with them so I built this one. It just uses the standard mail() function in php. Some things I picked up in my investigations.

  1. Setting the headers. With the header information I send, you have less chance of being blocked by spam blockers.

    Since I included the extra header information, I have not had ANY returns do to automated mail servers.
  2. The sleep timer. Using this stops most web servers stopping your email batch as the server may have rules on what they consider is spam sending. One of the popular rules, if you email hundreds and hundreds of emails in one go, you must be sending spam. The sleep counter just pauses the batch ( after 100 emails in this case ) waits 25 seconds and sends the next 100.
  3. ignore_user_abort(); just allows the batch to do its thing behind the scenes and you can carry on working.
    Lastly, just in case. You email body should contain NO javascript or it may be blocked. I batch email HTML invoices and account updates to customers.
    This worked for me anyway. Hope this helps
    <?php

    set_time_limit(0);

    ignore_user_abort();
    $from = '"';

    $from .= $_SESSION["company_name"];

    $from .= '"';

    $from .= '<';

    $from .= $_SESSION["company_email"];

    $from .= '>';
    $headers = 'MIME-Version: 1.0' . "\r\n";

    $headers .= 'Content-type: text/html; charset=iso-8859-1' . "\r\n";

    $headers .= 'From:' . $_SESSION["company_name"] . '<' . $_SESSION["company_email"] . '>' . "\r\n";

    $headers .= "X-Priority: 3\r\n";

    $headers .= "X-MSMail-Priority: Normal\r\n";

    $headers .= "X-Mailer: PHP / ".phpversion()."\r\n";

    $sleep_counter = 0;

    foreach($recipients as $email) // replace recipients with your $all_emails variable.

    {

    $to = $email;

    mail($to,$_SESSION["subject_line"],$_SESSION["customer_batch_email"],$headers);

    // I store my message in $_SESSION["customer_batch_email"], you can replace it with your $message variable.
    $sleep_counter +=1;

    if (($sleep_counter)== 100)

    {

    sleep(25);

    $sleep_counter = 0;

    }

    }

    ?>

M
mmponline author 3/30/2007

DaleM
Sorry for my late reply. I've been out for a few days.

I've reduced the code to the following without getting error messages, but the mails are not sent. When saving the record change, it takes quite a while, and then goes back to my list page. No messages are sent, however. Can you (or anybody) please assist. I need to get this working!!!

function BeforeEdit(&$values, $where)

{
global $conn;

$all_emails="";

$mailfrom1 = "info@verlorenkloof.co.za";

$strSQL = "select Email from Login where Available='Yes please'";

$rs = db_query($strSQL,$conn);

while ($data = db_fetch_array($rs))

$all_emails .= $data["Email"].", ";

$all_emails = substr($all_emails,0,strlen($all_emails)-2);

$message="The following rentals are available. Please book at our office. For more info on rental availability,login to your account on our website: www.verlorenkloof.co.za\r\n";;
$subject="Rental Availability:";
$message.= "From : ".$values["c_from"]."\r";

$message.= "To : ".$values["c_to"]."\r";

$message.= "Croft Number : ".$values["c_croftid"]."\r";

$message.= "Rate : ".$values["c_rate"]."\r";

$message.= "Beds : ".$values["c_beds"]."\r";

$message.= "Contact and Details :\r ".$values["c_admin"]."\r";
set_time_limit(0);

ignore_user_abort();
$headers = 'MIME-Version: 1.0' . "\r\n";

$headers .= 'Content-type: text/html; charset=iso-8859-1' . "\r\n";

$headers .= "X-Priority: 3\r\n";

$headers .= "X-MSMail-Priority: Normal\r\n";

$headers .= "X-Mailer: PHP / ".phpversion()."\r\n";

$sleep_counter = 0;
mail($all_emails, $subject, $message, 'From: '.$mailfrom1, $headers);
$sleep_counter +=1;

if (($sleep_counter)== 100);
sleep(25);

$sleep_counter = 0;
return true;
}
M
Martijn 3/30/2007

DaleM

Sorry for my late reply. I've been out for a few days.

I've reduced the code to the following without getting error messages, but the mails are not sent. When saving the record change, it takes quite a while, and then goes back to my list page. No messages are sent, however. Can you (or anybody) please assist. I need to get this working!!!

function BeforeEdit(&$values, $where)

{
global $conn;

$all_emails="";

$mailfrom1 = "info@verlorenkloof.co.za";

$strSQL = "select Email from Login where Available='Yes please'";

$rs = db_query($strSQL,$conn);

while ($data = db_fetch_array($rs))

$all_emails .= $data["Email"].", ";

$all_emails = substr($all_emails,0,strlen($all_emails)-2);

$message="The following rentals are available. Please book at our office. For more info on rental availability,login to your account on our website: www.verlorenkloof.co.za\r\n";;
$subject="Rental Availability:";
$message.= "From : ".$values["c_from"]."\r";

$message.= "To : ".$values["c_to"]."\r";

$message.= "Croft Number : ".$values["c_croftid"]."\r";

$message.= "Rate : ".$values["c_rate"]."\r";

$message.= "Beds : ".$values["c_beds"]."\r";

$message.= "Contact and Details :\r ".$values["c_admin"]."\r";
set_time_limit(0);

ignore_user_abort();
$headers = 'MIME-Version: 1.0' . "\r\n";

$headers .= 'Content-type: text/html; charset=iso-8859-1' . "\r\n";

$headers .= "X-Priority: 3\r\n";

$headers .= "X-MSMail-Priority: Normal\r\n";

$headers .= "X-Mailer: PHP / ".phpversion()."\r\n";

$sleep_counter = 0;
mail($all_emails, $subject, $message, 'From: '.$mailfrom1, $headers);
$sleep_counter +=1;

if (($sleep_counter)== 100);
sleep(25);

$sleep_counter = 0;
return true;
}


Check the smtp settings in php.ini. When you run your own mailserver check the limits of that.

When you use the smtp of your ISP ask them what they max allow or maybe you are banned already ..

Here in the Netherlands a lot of ISP started blocking outgoing smtp (port 25) traffic (+/- 1 month ago). You have to use the smtp of the ISP to send mail. Maybe this helps. gl
Martijn

D
Dale 3/30/2007

Stephan, I think you need to move the email snippet outside your while loop.
function BeforeEdit(&$values, $where)

{
global $conn;

$all_emails="";

$mailfrom1 = "info@verlorenkloof.co.za";

$strSQL = "select Email from Login where Available='Yes please'";

$rs = db_query($strSQL,$conn);

while ($data = db_fetch_array($rs))

$all_emails .= $data["Email"].", ";
} end your while loop as you now have all the email addresses you want to batch now.
I can appreciate the point about making code smaller. Most times once I get something working I leave it alone and get on with the next obstacle. I usally go back after a few months, more experienced gained, and then rethink the code or just cleanup.

But the snippet is important. Your additions or changes such as $message.= "From : ".$values["c_from"]."\r"; is not quite right. I believe the proper syntax should be $header.= etc etc. You have four or five lines like that. Then make sure you place them together with the other $header line.

Also the timing and placement of variables is important, so just have a review of my original snippet.
It takes about 10 to 15 secs for me to batch 300 emails, so it should not take that long. And the loop looking for all the email addresses should be very fast.

M
mmponline author 3/31/2007

DaleM
I've changed 3 entries in my Available field to Sure, then changed the code in the following line to:

$strSQL = "select Email from Login where Available='Sure'";


Thus, only 3 e-mails should be sent. Stil nothing comes through (these addresses are setup on my box for testing.
My original code sent the mails to few people, but not many. This code takes a while to go back to list page, but NO mails are sent at all.
There must be something wrong in the code...
I'm not sure how to make the changes you suggested, as I get a "undefiend variable error if I add a } after the line you suggested.
Thanks for your patience.
Stephan

D
Dale 3/31/2007

I believe you have some variables that were calling for values from the database. Looking at your snippet I may have been wrong in the point about the $headers and $message. I was assuming the TO and FROM were related to email addresses and not dates.
Am I assuming correctly that it will be the same message going to all emails for the guys who said Sure.
Leave the bottom } where it was, at the bottom. This is the closing loop for the Before Edit function.

Try adding some brackets around your while loop.
Find in your code the lines below,
while ($data = db_fetch_array($rs))

$all_emails .= $data["Email"].", ";

$all_emails = substr($all_emails,0,strlen($all_emails)-2);
and add some brackets as below
while ($data = db_fetch_array($rs))

{

$all_emails .= $data["Email"].", ";

}

$all_emails = substr($all_emails,0,strlen($all_emails)-2);
See if that stops the errors. IF you get errors the mail stuff will not email. So try the brackets and see how that goes. It should not take any time delays as you mention, but again if there were errors then it wont be working correctly. If it doesnt work, post exactly what you have and we can go from there.
Support may have a better answer for you. This is just something that I put together that works in my scenario.

M
mmponline author 4/2/2007

DaleM

I get no Error messages, but no mails are sent. You're right, the mails should only go to "Sure". I've 3 entries setup with sure. Once I get the message to go through, I'll try to add more, to see if it works. It takes about 20 seconds from hitting save untill the page returns to the list page. It seems to be long...
My code looks like this now:

function BeforeEdit(&$values, $where)

{
global $conn;

$all_emails="";

$mailfrom1 = "info@verlorenkloof.co.za";

$strSQL = "select Email from Login where Available='Sure'";

$rs = db_query($strSQL,$conn);

while ($data = db_fetch_array($rs))

{

$all_emails .= $data["Email"].", ";

}

$all_emails = substr($all_emails,0,strlen($all_emails)-2);

$message="The following rentals are available. Please book at our office. For more info on rental availability,login to your account on our website: www.verlorenkloof.co.za\r\n";;
$subject="Rental Availability:";
$message.= "From : ".$values["c_from"]."\r";

$message.= "To : ".$values["c_to"]."\r";

$message.= "Croft Number : ".$values["c_croftid"]."\r";

$message.= "Rate : ".$values["c_rate"]."\r";

$message.= "Beds : ".$values["c_beds"]."\r";

$message.= "Contact and Details :\r ".$values["c_admin"]."\r";
set_time_limit(0);

ignore_user_abort();
$headers = 'MIME-Version: 1.0' . "\r\n";

$headers .= 'Content-type: text/html; charset=iso-8859-1' . "\r\n";

$headers .= "X-Priority: 3\r\n";

$headers .= "X-MSMail-Priority: Normal\r\n";

$headers .= "X-Mailer: PHP / ".phpversion()."\r\n";

$sleep_counter = 0;
mail($all_emails, $subject, $message, 'From: '.$mailfrom1, $headers);
$sleep_counter +=1;

if (($sleep_counter)== 100);
sleep(25);

$sleep_counter = 0;
return true;
}


Thanks for your patience with this.

D
Dale 4/2/2007

We need a loop around the mail command to mail to each email address you have accumulated.

Im showing my suggesting in bold below. It should work for you.
Two things to note.

FIrst, I fire my snippet from the list page, here we are trying the BeforeEdit page. Im not a hundred percent sure if this may be the time delay cause.
Second, I see you are truncating the $all_emails variable, which I do myself to get rid of the last comma that might be in the field. But the $all_emails value must end up looking like the example below to work in the foreach loop.
test1@sample.com,test2@sample.com,test3@sample.com,test4@sample.com etc etc.
Give this a shot, hopefully it just all works.
[code]

function BeforeEdit(&$values, $where)

{
global $conn;

$all_emails="";

$mailfrom1 = "info@verlorenkloof.co.za";

$strSQL = "select Email from Login where Available='Sure'";

$rs = db_query($strSQL,$conn);

while ($data = db_fetch_array($rs))

{

$all_emails .= $data["Email"].", ";

}

$all_emails = substr($all_emails,0,strlen($all_emails)-2);

$message="The following rentals are available. Please book at our office. For more info on rental availability,login to your account on our website: www.verlorenkloof.co.za\r\n";;
$subject="Rental Availability:";
$message.= "From : ".$values["c_from"]."\r";

$message.= "To : ".$values["c_to"]."\r";

$message.= "Croft Number : ".$values["c_croftid"]."\r";

$message.= "Rate : ".$values["c_rate"]."\r";

$message.= "Beds : ".$values["c_beds"]."\r";

$message.= "Contact and Details :\r ".$values["c_admin"]."\r";
$headers = 'MIME-Version: 1.0' . "\r\n";

$headers .= 'Content-type: text/html; charset=iso-8859-1' . "\r\n";

$headers .= "From: ". $mailfrom1 ."\r\n"; //this could be done here or in the mail function itself like you had it.

$headers .= "X-Priority: 3\r\n";

$headers .= "X-MSMail-Priority: Normal\r\n";

$headers .= "X-Mailer: PHP / ".phpversion()."\r\n";
set_time_limit(0);

ignore_user_abort();

$sleep_counter = 0;

$recipients = array();

$recipients = explode(',', $all_emails);

foreach($recipients as $email)

{

$to = $email;

mail($email, $subject, $message, $headers);
$sleep_counter +=1;

if (($sleep_counter)== 100);
sleep(25);

$sleep_counter = 0;

}
return true;
}

[code]

M
mmponline author 4/2/2007

We're getting there - Good news and bad.
Good news. - The script now sends the mails as required. I first sent it to Sure (3 records only) and it went through. Just took about 35 sec. on an ADSL line.
Bad news - I then changed the sure to Yes (about 53 records). It took more than 10 min. to send. When I check the e-mails during the update process, however, all my setup acount's mails came through. Just not sure if the cleints received their mails. Will have to confirm tomorrow in daytime to see if they received it.
Unfortunately, it takes very long. It also seems like the mails are sent twiceto the same addresses. If there is a way to get the time down, it seems like this script would do the job.

The only change I made to the script is to take this line out:

$headers .= 'Content-type: text/html; charset=iso-8859-1' . "\r\n";


as I don't need the text to be in HTM format.
Everything seems to work fine now, accept for the VERY slow process of sending! My events have a return to page on the after record updated, thus nothing can go on before the update took place. Any suggestions to cut on this time?

D
Dale 4/2/2007

Glad to see its advancing a bit. All trial and error on my side for sure.

The one line you took out, should be left in. It specifies the message is in either text or html. So if you dont put any html code in your email body, it will still be recieved as text. (This has nothing to do with the speed, these headers just help not being blocked as spam)
$headers .= 'Content-type: text/html; charset=iso-8859-1' . "\r\n";
The sent twice is what gets me.

This must be because we have the snippet in the before edit. This means the emails might to be sent every time you open open the record to edit. THen if you save the first thing PHPRunner does it tell the edit page to display after saving. Poof email may be going out again. Im not absolutely sure about this one.
Personally I would put this snippet in the AFTER EDIT rather than the before EDIT. But then you will have to address those variables you are filling in the email body. You may want to capture the values in a session variable on the before edit event.
You might want to rethink when you send emails. No matter what event you place this code in, its going to send emails whenever you save an edit. What happens if you just save twice. Emails gone twice. You may want to set up a session variable to set when the batch runs. Something like that.
Im sorry I cant be much more help here. I have to burry my head into some code over the next couple of days. Deadlines to meet. Seems like that all I do nowadays, try and make deadlines.

Admin 4/4/2007

I think this is the time for me to jump in.
I'll try to keep this short - sending bulk emails from PHP script like this is not a good idea. When number of emails to be sent reaches a certain point it becomes a slow, unreliable procedure. You never be able to tell if all emails were sent successfully or timeout happened at some point. It also makes the person who performed the record edit to wait indefinitely without seeing any progress or having a chance to interrupt it. Besides that it makes hard to troubleshoot errors like email sent twice.
What I believe to be a better design is having a separate process (script) that checks some table with emails to be sent and sends some portion of it like 200-300 emails in one go. This script can be scheduled to run on timely fashion via cron job. After each run emails in that table will be marked as processed.
Btw, this forum software uses similar approach. When I send an email from the control panel I can see it being sent in some small chunks. Not really sure how exactly it was implemented however idea is the same.

D
Dale 4/4/2007

I agree with the Admin comment completely.
As I mentioned, I run the suggested script from my list page, where I actually click the button to send. No chance on sending twice unless I click the button twice and even then I have a confirm script in place.
Because it is from the list page, No waiting while the emails are being sent. It is all done in the background.
The emails are sent in 100 email chunks and I have a log of what has been sent if a failure occurs.
I had my thoughts that it might not be appropriate to use in the Edit events.
It would be a great addition to the software if a batch email process was incorporated. One where you could select an email template, and then select a query to give you the batch list of customers you wished to email. Complete with logging, setting up cycling times etc. etc. A lot of folks ask the question about batch emailing.
Just trying to help.